Prim.rts_heap_size() and Prim.rts_memory_size()

[cryptoschindler]

a) What exactly is the difference between rts_heap_size and rts_memory_size?

b) We are testing how big we can grow a Buffer in a Motoko canister that is serialized into an Array in stable memory in the preupgrade hook before we exceed the instruction limit. The Buffer is initalized from the Array in stable memory and in the postupgrade hook we reset the stable variables to free memory.

memory size is the result of a call to Prim.rts_memory_size and heap size the result of a call to Prim.rts_heap_size. The order of call is

• growBuffer
• memory_size
• upgrade canister
• heap_size
  for each index in the table.

Growing transaction size to 4000000...
Upgrading...
┌─────────┬──────────────┬────────────────────────────┬──────────────────────────┬───────────────────────────┬─────────────────────────┬────────────────────┐
│ (index) │   canister   │ memory size before upgrade │ heap size before upgrade │ memory size after upgrade │ heap size after upgrade │ upgrade successful │
├─────────┼──────────────┼────────────────────────────┼──────────────────────────┼───────────────────────────┼─────────────────────────┼────────────────────┤
│    0    │ 'copying-gc' │         '2.19 MB'          │ '0.7237930297851562 MB'  │        '17.69 MB'         │ '5.959255218505859 MB'  │        true        │
│    1    │ 'copying-gc' │         '4.88 MB'          │ '2.0765953063964844 MB'  │        '52.56 MB'         │ '17.86111068725586 MB'  │        true        │
│    2    │ 'copying-gc' │         '6.38 MB'          │  '4.254154205322266 MB'  │        '87.44 MB'         │ '29.76296615600586 MB'  │        true        │
│    3    │ 'copying-gc' │         '12.75 MB'         │  '7.533042907714844 MB'  │        '174.56 MB'        │ '59.51760482788086 MB'  │        true        │
│    4    │ 'copying-gc' │         '26.13 MB'         │ '15.255535125732422 MB'  │        '348.94 MB'        │ '119.02688217163086 MB' │        true        │
│    5    │ 'copying-gc' │         '57.13 MB'         │ '29.706295013427734 MB'  │        '697.56 MB'        │ '238.04543685913086 MB' │        true        │
│    6    │ 'copying-gc' │         '57.13 MB'         │ '38.922794342041016 MB'  │        '871.94 MB'        │ '297.55471420288086 MB' │        true        │
│    7    │ 'copying-gc' │         '98.56 MB'         │  '50.66817855834961 MB'  │       '1220.56 MB'        │ '416.5734405517578 MB'  │        true        │
│    8    │ 'copying-gc' │        '125.50 MB'         │  '76.66597366333008 MB'  │       '1743.56 MB'        │ '595.1012725830078 MB'  │        true        │
│    9    │ 'copying-gc' │        '214.94 MB'         │  '99.41592025756836 MB'  │       '2440.88 MB'        │ '833.1383819580078 MB'  │        true        │
│   10    │ 'copying-gc' │        '284.06 MB'         │ '149.36919784545898 MB'  │       '3486.88 MB'        │ '1190.1940460205078 MB' │        true        │
└─────────┴──────────────┴────────────────────────────┴──────────────────────────┴───────────────────────────┴─────────────────────────┴────────────────────┘

why is the heap size significantly bigger after the upgrade? for the memory size, i think it is because this number can only grow, but never shrink and throughout the upgrade process the canister somehow consumes this amount of memory. does it still hold true that we pay for Prim.rts_memory_size, not for the actual memory being consumed by the canister?
c) what is the best way to determine the actual memory used by the heap? what is the best way to determine the actual memory used by the stable memory?

[timo]

@cryptoschindler

The Buffer is initalized from the Array in stable memory and in the postupgrade hook we reset the stable variables to free memory.

It would be helpful, to be able to give a better answer, if you can provide the code. I can guess how you are doing it but I am not 100% certain about it.

So index 0-10 are independent experiments? Not building on each other (like grow, upgrade, grow, upgrade again, grow again, etc.), right? How large is the Buffer in experiment with index i and what is the data type of the elements?

The heap size after upgrade is bigger because the deserialization creates an additional copy of the data in the heap. First Motoko fills the stable var Array and then your postupgrade copies it into another Buffer. Not sure why it becomes 8x larger though as opposed to only 2x. Do you create the Buffer with initial capacity? Or do you let it grow naturally (geometrically by a factor of 1.5x). I suspect the latter because that will explain the additional allocations that will increase the heap (before GC runs, which didn’t happen yet in your experiments).

Just curious, what is the goal of the experiment?

[cryptoschindler]

This is how the Buffer is initialised from the stable variable after the upgrade

    private var _transactions : Buffer.Buffer<Types.Transaction> = Buffer.fromArray(state._transactionsState);

This is how the Buffer is grown

    public func grow(n : Nat): Nat {
      let token = Principal.toText(this);
      let buyer = AID.fromPrincipal(this, null);
      let time = Time.now();
      for (i in Iter.range(1, n)) {
        _transactions.add({
          token = token;
          seller = this;
          price = 1000;
          buyer = buyer;
          time = time;
        });
      };
      _transactions.size();
    };

In the postupgrade, transactionsState is set to an empty array.

Index 0-10 are independent experiments, the canister is reinstalled between the rounds.
The Buffer in index i contains the amount of transactions specified in the transactions column, you can see the structure of a transaction in the above grow method.

I understand that the deserialisation take up additional space, but I’m using the force garbage collectors as well with the same results and expected them to garbage collect after each message. As the calls after the upgrade are all separate messages, shouldn’t the extra copy in the heap be garbage collected by the time I call the methods to get the heap, memory and max_live_size?

In the the postupgrade really only empties the stable variable, the copying from stable variable to the Buffer happens with the initialisation of the replacement actor. The Buffer has no initial capacity.

The goal of the experiment is to asses under what conditions our canister won’t be upgradeable because of a Cycle limit exceeded error.

This is the run with live_heap_size and the force gb

Reinstalling...
Reinstalled
Growing transaction size to 2,800,000...
Upgrading...
┌─────────┬─────────────────┬──────────────┬───────────┬────────────┬────────────┬─────────────┬────────────────────┬──────────────────────┬──────────────────┬────────────────────┐
│ (index) │       gc        │ transactions │ reinstall │  max live  │    heap    │   memory    │ upgrade successful │ max live postupgrade │ heap postupgrade │ memory postupgrade │
├─────────┼─────────────────┼──────────────┼───────────┼────────────┼────────────┼─────────────┼────────────────────┼──────────────────────┼──────────────────┼────────────────────┤
│    0    │ 'copying-force' │   '40,000'   │   true    │ '1.39 MB'  │ '1.39 MB'  │  '3.44 MB'  │        true        │      '11.99 MB'      │    '11.99 MB'    │     '35.31 MB'     │
│    1    │ 'copying-force' │  '100,000'   │   true    │ '3.59 MB'  │ '3.59 MB'  │  '8.13 MB'  │        true        │      '29.95 MB'      │    '29.95 MB'    │     '87.81 MB'     │
│    2    │ 'copying-force' │  '400,000'   │   true    │ '14.01 MB' │ '14.01 MB' │ '28.63 MB'  │        true        │     '119.79 MB'      │   '119.79 MB'    │    '350.44 MB'     │
│    3    │ 'copying-force' │  '800,000'   │   true    │ '28.46 MB' │ '28.46 MB' │ '57.50 MB'  │        true        │     '239.57 MB'      │   '239.57 MB'    │    '700.69 MB'     │
│    4    │ 'copying-force' │ '1,000,000'  │   true    │ '34.58 MB' │ '34.58 MB' │ '69.75 MB'  │        true        │     '299.46 MB'      │   '299.46 MB'    │    '875.75 MB'     │
│    5    │ 'copying-force' │ '1,400,000'  │   true    │ '48.81 MB' │ '48.81 MB' │ '98.19 MB'  │        true        │     '419.24 MB'      │   '419.24 MB'    │    '1225.94 MB'    │
│    6    │ 'copying-force' │ '2,000,000'  │   true    │ '70.15 MB' │ '70.15 MB' │ '140.88 MB' │        true        │     '598.92 MB'      │   '598.92 MB'    │    '1751.25 MB'    │
│    7    │ 'copying-force' │ '2,800,000'  │   true    │ '99.11 MB' │ '99.11 MB' │ '198.75 MB' │        true        │     '838.48 MB'      │   '838.48 MB'    │    '2451.63 MB'    │
└─────────┴─────────────────┴──────────────┴───────────┴────────────┴────────────┴─────────────┴────────────────────┴──────────────────────┴──────────────────┴────────────────────┘

[timo]

The fact that max live equals heap seems to indicate that GC has just run, for otherwise they should differ. So your force GC seems to be in effect.

Does index i in your first post match index i in your last post, i.e. are they expected to have the same buffer size?

The heap size post upgrade is larger than pre upgrade because pre upgrade you only have the buffer (_transactions) on the heap and the stable array (state) is empty. After the upgrade you might have both full. You say " In the the postupgrade really only empties the stable variable". But how are you emptying it? Are you deleting the stable variable, setting it to the empty array within postupgrade?

In either case, I can’t explain why the heap is larger by a factor of 8. I would expect it to be less than 2x because the data isn’t really copied. At worst pointers are duplicated in _transactions and state, not the actual record.

[timo]

I think this is the reason: Before upgrade you have an array (inside Buffer) of transaction records. Each transaction is the same. The token fields in all those transactions are just pointers to the same Text. The Text (Principal.toText(this)) only exists once on the heap. The same happens to seller, buyer, time. But this “sharing” gets lost during serialization. After deserialization all the data is multiplied.

[cryptoschindler]

Does index i in your first post match index i in your last post, i.e. are they expected to have the same buffer size?

Yes

You say " In the the postupgrade really only empties the stable variable". But how are you emptying it? Are you deleting the stable variable, setting it to the empty array within postupgrade?

That’s exactly what I’m doing

Before upgrade you have an array (inside Buffer) of transaction records

Not sure why there is an array inside my Buffer .

I rewrote grow to this

    public func grow(n : Nat) : Nat {
      for (i in Iter.range(1, n)) {
        let token = Principal.toText(this);
        let seller = this;
        let buyer = AID.fromPrincipal(this, null);
        let time = Time.now();
        _transactions.add({
          token;
          seller;
          price = 1000;
          buyer;
          time;
        });
      };
      _transactions.size();
    };

And the new results look like this:

Reinstalling...
Reinstalled
Growing transaction size to 100,000...
Upgrading...
┌─────────┬─────────────────┬──────────────┬───────────┬────────────┬────────────┬─────────────┬────────────────────┬──────────────────────┬──────────────────┬────────────────────┐
│ (index) │       gc        │ transactions │ reinstall │  max live  │    heap    │   memory    │ upgrade successful │ max live postupgrade │ heap postupgrade │ memory postupgrade │
├─────────┼─────────────────┼──────────────┼───────────┼────────────┼────────────┼─────────────┼────────────────────┼──────────────────────┼──────────────────┼────────────────────┤
│    0    │ 'copying-force' │   '40,000'   │   true    │ '38.92 MB' │ '38.92 MB' │ '180.38 MB' │        true        │      '11.99 MB'      │    '11.99 MB'    │     '35.31 MB'     │
│    1    │ 'copying-force' │  '100,000'   │   true    │ '97.43 MB' │ '97.43 MB' │ '297.63 MB' │        true        │      '29.95 MB'      │    '29.95 MB'    │     '87.81 MB'     │
└─────────┴─────────────────┴──────────────┴───────────┴────────────┴────────────┴─────────────┴────────────────────┴──────────────────────┴──────────────────┴────────────────────┘
--------------------------------------------------

TBH I’m equally confused, why is the max_live smaller after the upgrade now .
Also I can’t even grow the Buffer to 400.000 transactions (heap of roughly 400MB), is this fishy for you or does it sound realistic?
Growing transaction size to 400,000... Error: Failed update call. Caused by: Failed update call. The Replica returned an error: code 5, message: "Canister r7inp-6aaaa-aaaaa-aaabq-cai exceeded the instruction limit for single message execution." unexpected error Error: Command failed: dfx canister call copying-force grow 20000

Why would I exceed the instruction limit when growing the Buffer?

[timo]

@cryptoschindler

Why would I exceed the instruction limit when growing the Buffer?

Try it with specifying an initial capacity of 400000 when you first create the _transactions Buffer. And then add to it like you do now. Then there will be less allocations happening, making both your code and the GC cheaper. See what size you can reach like that.

Generally I am not surprised that you hit the cycle limit. There are a lot of allocations happening.

[cryptoschindler]

I was under the impression that the Buffer size (except for when we have to resize) does not impact the cycle cost for equivalent insertion operations

[timo]

Right, that’s true. Just give it sufficient initial capacity and avoid the occasional resizing events because they are expensive.

[cryptoschindler]

EDIT: Buffer capacuty is increased by 1.5x

Is the Buffer size doubled if the limit is reached?
And do you have an explanation why after the upgrade the heap size is smaller than before?

[cryptoschindler]

EDIT: I made a temporary change to initialise the Buffer with a capacity of 400k, and I’m still not able to grow it beyond 280k entries… Why are calls exceeding the cycles limit after a certain size?

I can’t really give the buffer a sufficient initial capacity, as it’s always initialised from the stable variable, which initially is an empty array

[cryptoschindler]

BTW, I’m using moc-0.7.5 and the base library from this package set

[timo]

@cryptoschindler

EDIT: I made a temporary change to initialise the Buffer with a capacity of 400k

And did the numbers change in any way? Can you post the table for index 0,1 with this temporary change (maybe initial capacity 100k if index 1 is only 100k)? At least the memory size should change and be close to the heap size.

@cryptoschindler

And do you have an explanation why after the upgrade the heap size is smaller than before?

No.

@cryptoschindler

I’m still not able to grow it beyond 280k entries… Why are calls exceeding the cycles limit after a certain size?

Well, that’s just where the cycle limit is. 280k times the code in the for loop plus the memory allocation that comes with it reaches the limit. Nothing surprising about that.

[cryptoschindler]

@timo

Well, that’s just where the cycle limit is. 280k times the code in the for loop plus the memory allocation that comes with it reaches the limit. Nothing surprising about that.

I’m not calling grow with 280k as an argument, I grow the buffer incrementally by making subsequent calls to grow always passing 20_000 as an argument until I reach the desired maximum size. That’s why I’m confused, are insertions more costly the bigger the Buffer grows?

Even if I set the initial capacity of the Buffer to 400k and with the rewrite of grow to this

    public func grow(n : Nat) : Nat {
      for (i in Iter.range(1, n)) {
        let token = Principal.toText(this);
        let seller = this;
        let buyer = AID.fromPrincipal(this, null);
        let time = Time.now();
        _transactions.add({
          token;
          seller;
          price = 1000;
          buyer;
          time;
        });
      };
      _transactions.size();
    };

I’m not able to grow beyond ~280k entries.

Reinstalling...
Reinstalled
Growing transaction size to 100,000...
Upgrading...
┌─────────┬─────────────────┬──────────────┬───────────┬────────────┬────────────┬─────────────┬────────────────────┬──────────────────────┬──────────────────┬────────────────────┐
│ (index) │       gc        │ transactions │ reinstall │  max live  │    heap    │   memory    │ upgrade successful │ max live postupgrade │ heap postupgrade │ memory postupgrade │
├─────────┼─────────────────┼──────────────┼───────────┼────────────┼────────────┼─────────────┼────────────────────┼──────────────────────┼──────────────────┼────────────────────┤
│    0    │ 'copying-force' │   '40,000'   │   true    │ '40.29 MB' │ '40.29 MB' │ '183.00 MB' │        true        │      '1.53 MB'       │    '1.53 MB'     │     '25.56 MB'     │
│    1    │ 'copying-force' │  '100,000'   │   true    │ '98.43 MB' │ '98.43 MB' │ '299.25 MB' │        true        │      '1.53 MB'       │    '1.53 MB'     │     '58.81 MB'     │
└─────────┴─────────────────┴──────────────┴───────────┴────────────┴────────────┴─────────────┴────────────────────┴──────────────────────┴──────────────────┴────────────────────┘
--------------------------------------------------
Reinstalling...
Reinstalled
Growing transaction size to 400,000...
Error: Failed update call.
Caused by: Failed update call.
  The Replica returned an error: code 5, message: "Canister r7inp-6aaaa-aaaaa-aaabq-cai exceeded the instruction limit for single message execution."
unexpected error Error: Command failed: dfx canister call copying-force grow 20000
Error: Failed update call.
Caused by: Failed update call.
  The Replica returned an error: code 5, message: "Canister r7inp-6aaaa-aaaaa-aaabq-cai exceeded the instruction limit for single message execution."

    at checkExecSyncError (node:child_process:817:11)
    at execSync (node:child_process:888:15)
    at grow (/Users/moritz/projects/ic/flower-power-dao/power-equalizer/test-upgrade.js:21:13)
    at Object.<anonymous> (/Users/moritz/projects/ic/flower-power-dao/power-equalizer/test-upgrade.js:106:9)
    at Module._compile (node:internal/modules/cjs/loader:1105:14)
    at Module._extensions..js (node:internal/modules/cjs/loader:1159:10)
    at Module.load (node:internal/modules/cjs/loader:981:32)
    at Module._load (node:internal/modules/cjs/loader:827:12)
    at Function.executeUserEntryPoint [as runMain] (node:internal/modules/run_main:77:12) {
  status: 255,
  signal: null,
  output: [
    null,
    <Buffer >,
    <Buffer 1b 5b 33 31 6d 45 72 72 6f 72 3a 20 1b 28 42 1b 5b 6d 46 61 69 6c 65 64 20 75 70 64 61 74 65 20 63 61 6c 6c 2e 0a 1b 5b 33 33 6d 43 61 75 73 65 64 20 ... 180 more bytes>
  ],
  pid: 89917,
  stdout: <Buffer >,
  stderr: <Buffer 1b 5b 33 31 6d 45 72 72 6f 72 3a 20 1b 28 42 1b 5b 6d 46 61 69 6c 65 64 20 75 70 64 61 74 65 20 63 61 6c 6c 2e 0a 1b 5b 33 33 6d 43 61 75 73 65 64 20 ... 180 more bytes>
}

[timo]

I see. Actually I now see the 20,000 argument value in the logs further above. If you set the initial capacity to 400,000 then the cost for all insertions below that level must be the same. The only thing that gets more costly as the Buffer fills up is garbage collection. The number of objects that the GC has to look at is at least the 400k entries (even if they are null) and then 280k * 5 (for each of the five fields in your objects). That’s in the order of 2m. Then possibly more, but I don’t know the types of buyer and this. In other experiments I have hit the GC limit at 200-300m small object. You are still 100x below that. But your situation differs in these ways:

• larger size of the objects may contribute to higher GC cost
• maybe more objects hidden/nested in this
• you grow first by 20,000 and only have the remaining cycles for GC (I only ran GC and nothing before)

It may be helpful to do your experiment with less complicated records where you know the exact size, or even with primitive types instead of a record.

Your postupgrade numbers are way too small. Something must have gone wrong. The data cannot possibly fit in 1.53MB.

I still don’t understand why your memory size is so much larger than heap. Given the initial capacity of 400,000 I would expect heap and memory to be close to each other.

EDIT: You can try grow in smaller increments, like 1,000 instead of 20,000. Leaving a higher cycle budget for GC may let it succeed and you can grow the Buffer further.

[cryptoschindler]

@timo

but I don’t know the types of buyer and this.

buyer is of type Text, this of type Principal.

@timo

It may be helpful to do your experiment with less complicated records where you know the exact size, or even with primitive types instead of a record.

I could do that, but then my results aren’t really helpful for what I’m trying to achieve, right?

@timo

Your postupgrade numbers are way too small. Something must have gone wrong. The data cannot possibly fit in 1.53MB.

I realized that now that I initialize the Buffer with an initial capacity, I don’t populate it from the stable Array anymore and thus the state is just discarded after the upgrade. So that makes sense and I can ignore it until I figured out why I can’t grow the Buffer more

@timo

I still don’t understand why your memory size is so much larger than heap. Given the initial capacity of 400,000 I would expect heap and memory to be close to each other.

Maybe @claudio or @matthewhammer have some ideas?

@timo

EDIT: You can try grow in smaller increments, like 1,000 instead of 20,000. Leaving a higher cycle budget for GC may let it succeed and you can grow the Buffer further.

I did this previously with 5k, with the exact same results. But will try 1k now and see if something changes. Thanks for all your help, it’s really appreciated!

[matthewhammer]

@timo

But this “sharing” gets lost during serialization. After deserialization all the data is multiplied.

This is my expectation of the main issue as well.

@cryptoschindler

Not sure why there is an array inside my Buffer .

If you look at how Buffer is implemented (as Motoko code in base),
you will find a mutable array that holds its elements.

@cryptoschindler

      let buyer = AID.fromPrincipal(this, null);

How long are text values for buyer? (how big is it to serialize and deserialize?)

@cryptoschindler

@timo

It may be helpful to do your experiment with less complicated records where you know the exact size, or even with primitive types instead of a record.

I could do that, but then my results aren’t really helpful for what I’m trying to achieve, right?

There are enough complex factors now, the clarity about what is going wrong is already lost, right?

FWIW, I also strongly advocate what Timo is recommending, as an experiment to get more data about the issue. For instance, if you try using Blobs of a fixed size as a stand in for these records and if the same issue happens, then we know it has nothing to do with the complexities of the records in the buffer, and is more about the buffer implementation itself, and how it is used here. Without doing that experiment, I don’t see how to reduce the confusion further, unfortunately.

[matthewhammer]

@cryptoschindler

The order of call is…

Why not measure both at both points? (Why measure one, then wait and do stuff, and then measure the other?)

[claudio]

In ExperimentalStableMemory.mo, there is a function called stableVarQuery, that returns a query you can call to get the exact size of your stable variables in serialized format. it basically runs the pre-upgrade hook and then computes the size taken by stable variable data.

Eg:

actor {
  stable var state = "";
  public func example() : async Text {
    let memoryUsage = StableMemory.stableVarQuery();
    let beforeSize = (await memoryUsage()).size;
    state #= "abcdefghijklmnopqrstuvwxyz";
    let afterSize = (await memoryUsage()).size;
    debug_show (afterSize - beforeSize)
  };
};

ExperimentalStableMemory | Internet Computer Home

I haven’t had time to read the thread in detail, but as Timo and Matthew suggested, I strongly suspect that you are being hit by loss of sharing during stabilization. If your buffer entries contains duplicated references to the same Text value, Principal or Blob or immutable array or any other structured object each reference to the same unique object will get expanded to a copy of that object in the serialized format, leading to blow up atter deserialization. The only stuff that isn’t un-shared is the mutable data within, to preserve identity.

Although the array backing the buffer is only serialized once, any repeated references within its entries will still get copied.

This is a serious problem that ultimately is due to the fact that we (currently) use a mild extension of Candid to save stable variables, and Candid was not designed for that purpose.

We have some mitigations in mind, but the real solution (for us) is not to use Candid.

One possible workaround is to introduce separate tables to map each large Principal, Blob etc to unique small ids and then reference those entities in transaction records by their small id. You can then invert the tables and id when you need to decode the transactions.

[claudio]

This post describes the rts functions:

Motoko, Array, Memory

First of all, let me point out that constructing an array by repeated concatenation is not something you should do in production code. Since both operands need to be copied for each append, the cost of doing this is quadratic in the size of the final array. Assuming you want an immutable array, a more appropriate way is to create the complete array as mutable, initialise it via mutation, and then freeze it. As for the rts values: memory_size is the current size of the Wasm memory array (mostly…