Upgrades, Motoko, max size and Can I shrink it?

Developers Language Support
1

I had to do an upgrade today and I hit the default 3.2 max memory limit. I upped it a bit and still no dice. After sweating a bit and swapping out some Tries for Maps-v9 I figured out that snapshots were live(Thanks @berestovskyy and team! you made gave me much more peace today than I’ve had in past upgrades) and felt free to mess around a bit more.

Finally, I just upped the max memory to over 4GB:

dfx canister --network ic update-settings my_canister --wasm-memory-limit 4021225472

This worked but I saw some strange things.

Settings before upgrade:

Memory allocation: 0
Compute allocation: 0
Freezing threshold: 2_592_000
Memory Size: Nat(2364426472)
Balance: 3_593_596_313_527 Cycles
Reserved: 0 Cycles
Reserved cycles limit: 5_000_000_000_000 Cycles
Wasm memory limit: 3_821_225_472 Bytes
Module hash: 0x1bb62d741135b86b5df1ed58eec7300964757556311113aa2ac76c3619ea48b0
Number of queries: 2_440_269
Instructions spent in queries: 20_232_971_890_820
Total query request payload size (bytes): 5_742_780_745
Total query response payload size (bytes): 58_511_363_261
Log visibility: controllers

Note: I was a bit confused by the 2.3GB as we do have a ton of files in here but by my count, they should only be about 200MB.

During upgrade:

Memory allocation: 0
Compute allocation: 0
Freezing threshold: 2_592_000
Memory Size: Nat(5451802398)
Balance: 3_482_936_096_948 Cycles
Reserved: 0 Cycles
Reserved cycles limit: 5_000_000_000_000 Cycles
Wasm memory limit: 4_021_225_472 Bytes
Module hash: 0x427c7279b53457544fb421461299a79f6e7580eda842563895e36d988b67e345
Number of queries: 2_440_269
Instructions spent in queries: 20_232_971_890_820
Total query request payload size (bytes): 5_742_780_745
Total query response payload size (bytes): 58_511_363_261
Log visibility: controllers

Note: How did the memory get to 5451802398 when my limit was 4.0xxxx GB?

Now that it is running it is the same with an over 4GB Memory

Status: Running
Controllers: 5vdms-kaaaa-aaaap-aa3uq-cai a3lu7-uiaaa-aaaaj-aadnq-cai ahx36-fo6xi-5exvo-4xqpz-hyafh-54vpt-tor6v-tz2xh-vwfgx-kwc5u-6qe osqd3-qbnmc-cgcvo-mwxag-a5pcv-27dad-k67vk-ed45o-4yqe7-mvbaj-yae
Memory allocation: 0
Compute allocation: 0
Freezing threshold: 2_592_000
Memory Size: Nat(5485356830)
Balance: 3_481_216_290_053 Cycles
Reserved: 0 Cycles
Reserved cycles limit: 5_000_000_000_000 Cycles
Wasm memory limit: 4_021_225_472 Bytes
Module hash: 0x427c7279b53457544fb421461299a79f6e7580eda842563895e36d988b67e345
Number of queries: 2_440_295
Instructions spent in queries: 20_237_981_170_901
Total query request payload size (bytes): 5_742_787_947
Total query response payload size (bytes): 58_511_383_385
Log visibility: controllers

I ran the new motoko metrics(Very Very cool motoko team!) and I’m trying to figure out how I got to such a big allocation.

(
  record {
    heapSize = 395_138_868 : nat;
    maxLiveSize = 391_987_088 : nat;
    rtsVersion = "0.1";
    callbackTableSize = 256 : nat;
    maxStackSize = 2_097_152 : nat;
    compilerVersion = "0.12.1";
    totalAllocation = 3_781_262_476 : nat;
    callbackTableCount = 0 : nat;
    garbageCollector = "incremental";
    reclaimed = 3_386_123_608 : nat;
    logicalStableMemorySize = 1 : nat;
    stableMemorySize = 5_225 : nat;
    memorySize = 3_959_422_976 : nat;
  },
)

A heap size of 400MB is much more inline with what I’d expect. But why such a huge memory Size? Did I have an old upgrade that ran and was big and now every upgrade it just fudges a bit to make sure it has room in stable variable space in case the whole allocation is full of Motoko stable variables?

Finally, is there any way to shrink the allocation?

Second finally, will that callbackTableCount give me outstanding calls keeping my canister from shutting down?

cc @luc-blaeser @claudio @ggreif

1 Like
2

Hi Austin,
Thank you very much for informing us about this problem.

It would be interesting to know what the heap size was before the the upgrade. I also wonder whether you maybe use additional Motoko flags apart from --incremental-gc and/or particular upgrade options? I assume you use moc 0.12.1?

I have a theory, that might explain parts, but not everything:

  1. The heap size was really large before hitting the soft limit (3.2 GB). Also, internally, the Motoko GC schedules more aggressively just past 3.2 GB, so maybe there was some substantial garbage in the heap as well. The incremental GC will then allocate up to 4 GB to perform its task (copying scratch space).
  2. The (classical) Motoko upgrade had to copy the stable reachable objects (stable vars) to the stable memory. This was 326 MB of space. I believe the canister memory size includes both main memory and stable memory, this is why it can go beyond 4 GB. Maybe it includes additional internal memory, but I am not sure. I need to check with the IC runtime people.
  3. After the upgrade, the main memory should shrink back to the size of the stable objects plus the serialized stable data. Why this is not the case here, is the main question. I assume this size was the immediate state after the upgrade? I wonder whether there are maybe larger temporary allocations in the actor’s initialization/post-upgrade logic?

Unfortunately, there is no option to shrink main memory on the fly (this is also not foreseen in the current WebAssembly standard), although that would be nice to have.
Practically, there exist two possibilities for shrinking main memory on the IC:

  • Upgrade with classical persistence - which did not work here, and we need to figure out.
  • Expert mode: Upgrade with EOP but using graph copy - and be extremely careful that you used graph copy, upgrade with wasm_memory_persistence: opt Replace .

I am happy to analyze the problem if you could share code or have a repro.

3

I will send you the info in pm.

4

I did some experiments with toy a canister and can’t reproduce a problem, both with incremental and default gc.

I used the following code that allocates .5 GB of stable memory and creates a .5GB blob in a stable variable.

The main memory grows to ca. 1GB after upgrade but stays at 1GB on subsequent upgrades (which is what I would expect).

Adjusting the limit had no effect.


import Prim "mo:prim";

actor {

   stable var version = 0;
   stable var stats_calls = 0;
   version += 1;
   func getStats() : Text {
     debug_show ({
         version = version;
         rts_memory_size = Prim.rts_memory_size();
         rts_heap_size = Prim.rts_heap_size();
         rts_stable_memory_size = Prim.rts_stable_memory_size();
         stats_calls;
         dummy = "g";
       })
    };
    let blobSize = 512 * 1024 * 1024;
    let stablePageSize = 64 * 1024;
    if (Prim.stableMemorySize() == 0) {
      ignore Prim.stableMemoryGrow(Prim.natToNat64(blobSize / stablePageSize));
    };

    stable var blob = Prim.stableMemoryLoadBlob(0, blobSize);
    Prim.debugPrint (getStats ());

    system func preupgrade() {
    };

    system func postupgrade() {
    };

    public func stats() : async Text {
     stats_calls += 1;
     getStats() };

};
1 Like
5

Hey Austin,
The canister status memory_size is an unfortunate naming. In fact it returns memory_usage as mentioned in the specification.

It’s defined as:

    pub fn memory_usage(&self) -> NumBytes {
        self.execution_memory_usage()
            + self.canister_history_memory_usage()
            + self.wasm_chunk_store_memory_usage()
            + self.system_state.snapshots_memory_usage
    }
6

Ahhhh…so did having a snapshot maybe torpedo me here? I think the snapshot is like 1.1 GB.

In that case it was safe to go over the 4GB minimum because the motoko and upgrade engines were likely ignoring snap size for safety checks?

7

Thanks for pointing out - just saw in the 12.1 release notes

@luc-blaeser is __motoko_runtime_information() automatically added to all Motoko canister APIs now after 0.12.1?

1 Like
8

The “Wasm memory limit” is a Wasm memory limit only, i.e. it’s a limit for Prim.rts_memory_size();. It does not limit snapshots, nor stable memory.

9

Yes, but it explains why the DFX status was showing a memory size of 5.2 GB when I had limited the memory to 4.02 GB.

The DFX status is showing memory usage and not wasm memory size.

I know you guys live and breathe this every day, but that is very confusing for someone we would consider themselves a substantially informed user.

It would be great if DFX status could separate those out for you. And even better if it could detect, it was a Motoko canister and throw those variables in as well. Or it would be helpful to have some of that information in the error message about the failed upgrade.

Probably just something to throw on the backlog.

1 Like
10

Hi Byron,

Yes, __motoko_runtime_information() is available on all Motoko canisters with version >= 0.12.1. Of course, the call is restricted to controllers only.

1 Like