Officializing the Incremental GC in Motoko

[icme]

@bitbruce

we were using trie to store small fine-grained data, and after storing 700,000 records, the memory footprint was 1.5 G.

I’m surprised that you didn’t hit issues with the compacting GC before 1.5GB

@bitbruce

Then the issue arose, continued to write data to the canister (no delete operations)， about 100,000 records, and then the memory growth rate increased significantly, reaching the 4G limit in a few hours.

Is your heap size actually 4GB when you call Prim.rts_heap_size()? Or are you referring to your canister’s memory size Prim.rts_memory_size()?

@bitbruce

During this process, it seems that the gc cannot catch up with the speed of data writing, resulting in a large amount of garbage buildup. Am I understanding this correctly?

If you’ve gotten to the point where the GC is blocked (too much garbage to collect over a few rounds of DTS consensus), then you’re in danger of your canister becoming unresponsive to additional insertions. This is where the incremental GC should shine (allowing you to use the full 4GB heap).

You should be able to activate the incremental GC before the 0.15.3 dfx release by following these instructions - Incremental Garbage Collector Beta Release: Motoko Update Part 12

One additional area of improvement suggestion I might recommend. The TrieMap in motoko base isn’t nearly as memory efficient as some of the other libraries out there. Collection libraries | canister-profiling has statistics on a few of the currently available map libraries out there. Maybe look into a data migration that can save you space while you’re at it!