Motoko canister memory size increased after upgrade

Here is the Motoko canister code:

import HashMap "mo:base/HashMap";
import Principal "mo:base/Principal";
import Iter "mo:base/Iter";
import Array "mo:base/Array";

actor {
	private stable var mapEntries : [(Principal, Nat)] = [];
	private var map = HashMap.HashMap<Principal, Nat>(0, Principal.equal, Principal.hash);
	public shared(msg) func addItem(n: Nat) {
		map.put(msg.caller, n);	
	};

	public query func test(): async Bool {
		true
	};

	system func preupgrade() {
        mapEntries := Iter.toArray(map.entries());
    };

    system func postupgrade() {
        map := HashMap.fromIter<Principal, Nat>(mapEntries.vals(), 1, Principal.equal, Principal.hash);
        mapEntries := [];
    };
};

Install the canister and check the status, memory size is 378007:

Canister status call result for test.
Status: Running
Controller: tfuft-aqaaa-aaaaa-aaaoq-cai
Memory allocation: 0
Compute allocation: 0
Freezing threshold: 2_592_000
Memory Size: Nat(378007)
Balance: 4_000_000_000_000 Cycles
Module hash: 0xd9cac7fa14832eec53dbaf56b166216976a418d94b8795e4ba50bbe15bbd7b02

Insert 100 entries into the map and check the status, memory size is 443543, increased 65536, until now, it all make sense:

Canister status call result for test.
Status: Running
Controller: tfuft-aqaaa-aaaaa-aaaoq-cai
Memory allocation: 0
Compute allocation: 0
Freezing threshold: 2_592_000
Memory Size: Nat(443543)
Balance: 4_000_000_000_000 Cycles
Module hash: 0xd9cac7fa14832eec53dbaf56b166216976a418d94b8795e4ba50bbe15bbd7b02

Then I removed the useless query function and upgraded the canister:

// removed this function
public query func test(): async Bool {
	true
};
// then upgrade:
// dfx build test
// dfx canister install test -m=upgrade

But the memory size increased to 639424, it does not make sense to me:

Canister status call result for test.
Status: Running
Controller: tfuft-aqaaa-aaaaa-aaaoq-cai
Memory allocation: 0
Compute allocation: 0
Freezing threshold: 2_592_000
Memory Size: Nat(639424)
Balance: 4_000_000_000_000 Cycles
Module hash: 0xae4d9d217a2e1ffa29e03ff6fa999d66805ac3037dee9222a616d29cec0f252c

I deleted one function, the wasm module size should be smaller, but instead, the canister memory size increased, why?
I’m thinking maybe it has something to do with the Motoko upgrade functions? Maybe the mapEntries memory is not cleared after the upgrade? Will Rust canisters have the same problem?

I’m using dfx 0.8.0.

1 Like

This is not unexpected: For the upgrade, the Motoko runtime serializes your whole state to stable memory, in a format that is almost like Candid. This is not implemented in the most efficient way yet; in particular it will

  1. increase the main memory to have space to assemble the candid data.
  2. serialized the data in Candid format into that space
  3. copy it in bulk to stable memory
  4. (now the upgrade happens)
  5. increase the main memory to have space for the encoded candid data
  6. copy it in bulk from the stable memory
  7. decode from Candid into Motoko

The scratch space used in step 5 will now be unused, and reclaimed by the GC, but WebAssembly doesn’t allow code to give back memory to the system, so the memory footprint will appear large.

But Motoko will use that memory for new data before it grows the memory again, so it is altogether not too bad.

(The code change is a red herring, I assume you’d see the same behavior even if you upgrade to the same code.)

4 Likes

Thanks for making it clear:)

Perfect answer!

and Rust canister has anything different? Will the Rust canister increase memory size after upgrade?

In rust you manage stable memory manually, so this effect will only be visible if you program it that way.

So in most cases I don’t expect a rust cansiter to exhibit unexpected memory consumption after an upgrade.

2 Likes