Today in the IC developer discord community call, a developer asked how they can guard canisters against hitting canister memory limitations (and becoming zombies/unresponsive).
There are several solutions for preventing this from happening, but I’m curious why in the first place canisters were designed without memory limit safeguards in mind, and why protocol-level setting such as
freezing_threshold for cycles doesn’t exist for heap/stable memory storage limits. Such a protocol level feature could automatically reject update calls when certain heap/canister memory cliffs are about to be breached.
I’ll post some of my thoughts below regarding how developers can currently combat storage limits, but I’d love to hear more about different approaches that developers in both the Motoko and Rust community are using to safeguard their canisters from heap/stable memory limitations.
Hopefully discussions like this will help inform common patterns, libraries, and new features that can protect current and future IC developers from unexpected storage capacity issues.
Outside of a protocol level feature provided by DFINITY or a auto-scaling canister data storage framework like CanDB, I’ve found that a great way to guard against an unresponsive canister is to set canister heap/total memory limits for every canister that you spin up. In Motoko, I’ve done this by utilizing the
Prim.rts_heap_size() method on every update API call (or every x calls) in order to retrieve and then compare these memory stats against the limits I’ve set before inserting more data or throwing my own
MemoryLimitExceeded error. In Rust, there are similar methods one can use, for example this open source example by CanisterGeek.
Once this limit gets hit, instead of the canister “locking up” completely, only subsequent update calls will throw errors. A developer can then take the necessary time to decide on a temporary and/or permanent solution for moving forward, but at least the canister and data inside is not lost.