Dear community,
in few weeks we will propose a replica change that will initialize the new wasm_memory_limit field of canister settings to 3GiB by default (the actual value depends on the current memory usage and is explained below). That will effectively reduce the Wasm memory available to a canister from 4GiB to 3GiB.
If you know about the risks of reaching 4GiB and have programmed your canister carefully to use the entire 4GiB of Wasm memory, then you can opt out of the change by setting wasm_memory_limit=4GiB in canister settings using dfx version 0.20.1-beta.0 (or higher).
Otherwise, please read on to understand the upcoming change.
Motivation
The reason why we are proposing this change is to protect canisters from getting bricked. As you might know, the Wasm memory is 32-bit, which means that it cannot grow beyond the hard limit of 4GiB.
If a canister stores user data in the Wasm memory (instead of the stable memory), then its memory usage will increase with the number of users. Eventually, such a canister could reach 4GiB at which point it wonât be able to allocate more memory and will stop working. It is also likely that the pre-upgrade hook will fail to allocate, which will make the canister and its data unrecoverable.
Even if the canister doesnât store user data in Wasm memory, its memory usage may increase due to memory leaks.
In order to avoid such catastrophic scenarios, we implemented a soft limit for the Wasm memory. When the limit is set and the canister exceeds it, then messages will start failing. This will alert the developer to the potential memory issue and allow them to safely upgrade the canister to a version that uses less memory.
See also the original proposal thread and the NNS motion proposal.
How to set the limit?
The new field is supported in dfx version 0.20.1-beta.0 and versions that come after it. The mainnet also supports it.
You can query the current value of the limit for you canister using the dfx canister status command:
> dfx canister status <canister-id>
Canister status call result for <canister-id>.
Status: Running
...
Wasm memory limit: 0 Bytes
...
It most likely will be 0 meaning that it is not initialized and the limit is not enforced.
You can set the limit using the dfx canister update-settings command:
> dfx canister update-settings <canister-id> --wasm-memory-limit 3221225472
That sets that limit to 3GiB which can be confirmed with dfx canister status:
> dfx canister status <canister-id>
Canister status call result for <canister-id>.
Status: Running
...
Wasm memory limit: 3_221_225_472 Bytes
...
How will the limit be initialized by default?
In few weeks we will propose a replica change with the following logic for each canister:
-
if
wasm_memory_limitis already initialized, then do nothing for this canister. -
otherwise:
- if Wasm memory usage is below
2GiB, then set the limit to3GiB. - otherwise, set the limit to the midpoint between the current usage and
4GiB:(usage+4GiB)/2.
- if Wasm memory usage is below
How to opt out?
If you want to keep todayâs behavior, then you can set the limit to 4GiB (which is the current hard limit for 32-bit Wasm canisters). Be advised though, that if your canister reaches the hard limit, then it can potentially stop working and become unrecoverable.
How is the limit enforced?
The Wasm memory limit is a soft limit because it is not enforced for all message types. Currently, it is enforced only in update, init, and post_upgrade calls if they perform the memory.grow() operation. There is a pending replica change to enforce the limit in these calls even if they donât grow memory.
Here is why we chose to not enforce the limit for other message types:
heartbeat/timer: the limit is tentatively not enforced. It will be enforced after the canister logging feature ships. Without canister logging, it is difficult to debug failures in heartbeats and timers because the developer doesnât get the error message.query: queries are read-only, so even if they allocate memory, that allocation will be reverted when the query finishes. Also, developers that use queries to fetch canister data for backups should be able to do so even if the memory usage exceeds the limit.response callback(code that runs afterawaitpoint): it is difficult for developers to meaningfully recover from traps in response callbacks because some canister changes have already been committed before theawaitpoint. Introducing a new failure mode due to the Wasm memory limit, could lead to hard-to-debug issues.pre_upgrade: the limit is not enforced to allow upgrading the canister that is above the limit to a new version that reduces the memory usage.
Example error message
If the limit is enforced and the canister exceeds it, then the user will see the following error message:
Canister exceeded its current Wasm memory limit of 1000000 bytes. The peak Wasm
memory usage was 1310720 bytes. If the canister reaches 4GiB, then it may stop
functioning and may become unrecoverable. Please reach out to the canister
owner to investigate the reason for the increased memory usage. It might be
necessary to move data from the Wasm memory to the stable memory. If such high
Wasm memory usage is expected and safe, then the developer can increase the Wasm
memory limit in the canister settings.
What to do if you see the error
Please consider it as the proverbial canary in the coal mine. It warns about the underlying problem that the Wasm memory usage of the canister is growing for some reason. It could be due to a memory leak or a mis-design that stores all user data in Wasm memory instead of using the stable memory.
- Understand why the memory usage is growing. This is the most difficult step and there is no general recipe for it. It most likely requires inspecting the source code and finding data structures that can grow large.
- Back up the canister data if possible (e.g. if your canister has query endpoints that allow fetching the data).
- Upgrade the canister to the version that fixes the memory issue. For example, by moving large objects to the stable memory or by fixing the memory leak.