Increased Canister Smart Contract Memory

Roadmap
127

Now, Canister has 8GB stable memory and some low-level system api to use them, such as stable64_xxxx().

The typical way to use stable memory today is to serialize the state in the wasm heap into bytes before upgrading, and then overwrite it into stable memory. After the upgrade, restore to the wasm heap state from stable memory. And because the state in the wasm heap is often serialized as a whole, developers can only use 2GB in the wasm heap even in Rust, (the other 2GB is used to temporarily store serialized copies), and can only use 2GB to stable memory. And I doubt that a 2GB copy can be done within one block time (1s). Moreover, I actually encountered a situation where the canister on the mainnet could not be upgraded, but could be reinstalled, and it was difficult to find the problem.

As far as I know, here some canisters or tools trying to use more stable memory:

  1. The Internet Identity. the user anchors’ data is the only state which needs to be persisted, and will grow dynamicly. By determining the maximum space that each user anchor can occupy, it can be determined that the data of the newly added user anchor should be stored in the offset position of the stable memory.

  2. StableBTreeMap in Canisters, I tried to use it, but it’s a bit tricky to use when the situation gets complicated.

  3. Ic-stable-memory rust library, Good idea, but need to reimplement common data structures such as HashMap, Vec and more.

I think the perfect way to use stable memory is similar to using wasm heap. Wasm heap is also a stack-based linear memory space. We don’t need to consider these low-level interfaces when we use it. Why do we need to implement a memory allocator and memory manager when we use stable memory? This way we can directly use the lots of libraries in std. I’m looking forward to Dfinity’s official implementation of such a tool.

And, when I re-read the whole post, I found that wasm has the concept of multiple memories and stable vars, I think they are what I want, to be able to use stable memory like wasm heap. Unfortunately, it seems that neither of these features will make it into production anytime soon.

1 Like