Strategy on the upper limit of stable variable storage

Developers Language Support
1

In the relevant IC documents, I know that the upper limit of IC heap memory is 4GB.

  1. At the same time, I found relevant suggestions in your forum to
    “prevent the upgrade from becoming a brick, and the actual use should be controlled within 2GB”

I know that when upgrading ordinary variables, the hooks preupgrade and postupgrade need to be used to convert storage.

  1. However, our code completely uses stable variables to store business data, so there is no need for hook conversion storage during upgrade.
    So, in this case, can the storage limit be close to 4G?

In addition, in motoko, should Prim.rts_memory_size() or Prim.rts_heap_size() function be used to determine the current storage amount?

Thank you!

2

Top, looking forward to your reply!

3

Hi Haida,

You’re correct, the wasm32 heap memory limit is 4 GiB. The 2 GiB threshold is also relevant if you’re storing your state in the heap. You’ll need sufficient free memory to serialize the state from the heap to stable memory and back.

I’m not a Motoko expert, so perhaps @ggreif or @claudio could provide insights on the second point regarding stable variables and Prim functions.

4

Thank you and hope to receive a further comprehensive reply.

5

Thank you, Haida, for asking the question and apologies for the late response as I was sick last week.

In Motoko, you can use Enhanced Orthogonal Persistence with the --enhanced-orthogonal-persistence compiler flag. This enables Wasm64 and super-fast upgrades of constant time regardless of the heap size, without any use of stable memory. (Just in case you are interested in the technical aspects, here is a good reference: Smarter Contract Upgrades with Orthogonal Persistence | Proceedings of the 16th ACM SIGPLAN International Workshop on Virtual Machines and Intermediate Languages)

For Wasm64, the IC currently supports 6 GB and we intend to extend this further to 500 GB in future, to the same capacity as stable memory.

We also plan to make enhanced orthogonal persistence (EOP) the default for Motoko very soon.

The current default, classical persistence only supports maximum 2GB of stable objects (reachable through stable variables). And even then, several data structures (with sharing of immutable objects or structures with long pointer traces) do not scale at all and even crash at a few kilo- or megabytes.

Please let me know if I can provide more information.

1 Like
6

Hello, thanks for your reply!
You mean Wasm64/6GB, it is not supported yet, right?
When and what version do we need to wait?
At the same time, can Wasm64 handle more update calls per second?

At present, the rapid growth of our users requires us to pay attention to performance and capacity!

7

Hi Haida,
Thanks for following up.

Wasm64 is enabled since Sep 2024, and it has recently been extended to 6GB on mainnet.
I recommend using latest stable dfx 0.25.0 to use the latest fixes and improvements. Motoko’s enhanced orthogonal persistence (EOP) is also available since Sep 2024 and automatically adapts to increased Wasm memory sizes, e.g. extension to 6GB, without the need for any recompilation.

Wasm64 only targets memory scalability, while EOP additionally aims at speeding up upgrades for large canisters. Update call throughput still remains the same due to the actor model of the canister, executing messages only sequentially inside the canister.

There are specific architectures (balancing across multiple canisters, using query calls WITH certification for read-only-intense use cases) that may allow more throughput, however, with the price of extra software complexity.

8

OK, thanks. I am now preparing to compile and upgrade. Do I have to add "args" : "--enhanced-orthogonal-persistence"?

{
  "canisters": {
    "delta": {
      "main": "your-program.mo",
      "type" : "motoko",
      "args" : "--enhanced-orthogonal-persistence",
    },
"dfx": "0.25.0"
}

Also, does the program code remain unchanged?

9

Is there a system function that can query the current container storage limit so that we can be more intuitive?