Currently, Smart Contract Canister stable memory storage is capped due to Wasm limitations at 4 GB per Canister. To improve scaling, a new system API will be offered to Canisters that allow them use as much memory as available on their subnet (currently 300 GB).
- Community approved the design via NNS
- NNS proposals with code updates coming soon
- Ask questions
- Propose ideas
- Community Conversation - Increased Storage: August 25, 2021, 11 AM PDT / 8 PM CEST
- Draft plan for feature posted on the forum for review: August 26, 2021, 15:00 UTC
- NNS Motion Proposal submission: September 1, 2021, 15:00 UTC
- NNS Motion Proposal expiration: September 3, 2021, 15:00 UTC
- [NNS Motion proposal passed
- NNS proposals that upgrade the network’s code (as per design which passed Sep 3): Week of Sep 13
Currently, a canister on the IC has two types of storage available to it:
- A wasm heap which is constrained to 4 GiB because currently wasmtime does not support the wasm64 specification and hence has 32 bit addressing.
- A stable memory which is also currently constrained to 4 GiB because it too only has 32 bit addressing.
So a canister can under normal conditions store 8GiB of storage. However, when a canister is upgraded, its wasm heap is wiped so for all practical purposes, it only really has access to 4GiB of storage in the stable memory. In the past we demonstrated a proof of concept of BigMap which is a solution to enable an application to scale its storage by sharding its data across multiple canisters.
Based on discussions with external developers and with developers within the DFINITY foundation, we made the following observations:
- For a lot of applications that developers are currently trying to build, the 4GiB of a single canister is not quite enough. However, the capacity of a single subnet (300 GiB) is sufficient for the time being.
- BigMap is a good solution to scale storage to the capacity of multiple subnets but is not ready for use in production yet.
- It also appears that BigMap might be too heavy-handed an approach for scaling to the storage capacity of a single subnet and other mechanisms could be designed which are simpler to build and simpler to use.
Based on the above observations, the goal of this feature is to enable canisters to scale to the capacity of a single subnet by expanding how much stable memory it can store. At a high level, this will be done by offering a stable memory API that enables 64 bit addressing thereby allowing canisters to address 16 Exabytes of stable memory storage (probably more storage than what will ever be available on a single subnet). The feature also involves investigations into and making sure that the current data structures used for managing the stable memory of the canister can scale appropriately when they store a lot more than 4GiB of storage.