[Discussion] Approaches for preventing canisters from hitting memory limits

Developers
1

Today in the IC developer discord community call, a developer asked how they can guard canisters against hitting canister memory limitations (and becoming zombies/unresponsive).

There are several solutions for preventing this from happening, but I’m curious why in the first place canisters were designed without memory limit safeguards in mind, and why protocol-level setting such as freezing_threshold for cycles doesn’t exist for heap/stable memory storage limits. Such a protocol level feature could automatically reject update calls when certain heap/canister memory cliffs are about to be breached.

I’ll post some of my thoughts below :point_down: regarding how developers can currently combat storage limits, but I’d love to hear more about different approaches that developers in both the Motoko and Rust community are using to safeguard their canisters from heap/stable memory limitations.

Hopefully discussions like this will help inform common patterns, libraries, and new features that can protect current and future IC developers from unexpected storage capacity issues.




Outside of a protocol level feature provided by DFINITY or a auto-scaling canister data storage framework like CanDB, I’ve found that a great way to guard against an unresponsive canister is to set canister heap/total memory limits for every canister that you spin up. In Motoko, I’ve done this by utilizing the Prim module’s Prim.rts_heap_size() method on every update API call (or every x calls) in order to retrieve and then compare these memory stats against the limits I’ve set before inserting more data or throwing my own MemoryLimitExceeded error. In Rust, there are similar methods one can use, for example this open source example by CanisterGeek.

Once this limit gets hit, instead of the canister “locking up” completely, only subsequent update calls will throw errors. A developer can then take the necessary time to decide on a temporary and/or permanent solution for moving forward, but at least the canister and data inside is not lost.

10 Likes
2

Tagging a few prominent developers/projects in the community for input and approaches they currently take to guard against their canisters hitting storage capacity limitations.

@rckprtr @dymayday @hpeebles @Sherlocked @skilesare @dostro @yrgg @infu @quint @lastmjs

3

@brutoshi @neeboo @apotheosis, as well as some members of the DFINITY team @johan @ielashi @ulan

Sorry for the spam everyone!

4

Hey thanks for the ping !

At distrikt we have multiple ways to do it :

  • A glocal size counter that is the sum of each data added to the DB, with a hard cap around 4GB that prevent any further data addition.
  • Tracking the wasm page memory allocation.

Solution 1 is more accurate as the wasm memory allocation can sometimes be the result of greedy computation and not “garbage collected” until the next canister deployment.

One could probably use a guard to do it inspect_message in ic_cdk_macros - Rust

I’m curious to see other solutions :slight_smile:

Cheers

1 Like
5

How do you track the size of the data added? I’m assuming you’re including not just the size of the data, but also the increased size as a result of that data being inserted into the underlying data structure.

6

For each data to be added to the canister, I compute the size of the object with some tricks here and there using the Rust’s mem module and add the result to the global total memory size tracker.

1 Like