Motoko Serialization

To back up the data in the stable memory I would just take the entire memory pages and ship them off as Blob to another canister.

To clarify, do you mean to use the loadBlob function?

You should be able to send at least 2 MB in a single inter-canister message. With the stable memory being organized in memory pages of 64kB you can then ship 32 pages at once.

Oh interesting, I thought the 2 MB message size limit only applied to ingress messages. I didn’t know they also applied to inter-canister messages.

Thank you @timo for your comprehensive explanation. This gives me a lot to think about.

Due to the experimental warning, I have intentionally left out ExperimentalStableMemory from the solution. However, I am very hopeful about leveraging this library in the future. I have a few questions on that topic if you have the time.

  1. It sounds like the only way to get a blob from an instance of a complex type is to custom serialize it to a string format like JSON and then call Text.encodeUtf8. Is my understanding correct?
  2. If I need to restore all pages of stable memory after an upgrade to some HashMaps, could I still run into the cycle limit?
  3. To echo @jzxchiang, does that 2 MB message size limit apply to inter-canister messages?

Certainly not the only way. What complex type are you thinking about? You can probably serialize it directly to binary.

Theoretically, yes. If that happens then you could spread building the HashMap over several messages. The upgrade could leave the canister in a state in which only a post-upgrade function can be called and no other functions. And then you can call the post-upgrade function multiple times and each time you call it a few more entries get added to the HashMap. There are also ways for a canister to call itself in a loop, then it would be automated.

As far as I know, yes. Inter-canister messages end up in blocks and there is a block size limit so there must be a message size limit, too.

Thanks again. I like your idea to copy the stable data with multiple operations after the upgrade is finished. I’ll do some testing for the message size limit since the IC code is a bit over my head.

If you could show me how to do that, it would solve my biggest problem. Here are some examples of what I would call complex types:

public type Address = {
   id: Nat32;
   street1: Text;
   street2: Text;
   city: Text;
   province: Text;
   postalCode: Text;
   country: Text;
};

public type Person = {
   id: Nat32;
   firstName: Text;
   lastName: Text;
   emails: List<Text>;
   addresses: List<Nat32>;
};

@timo @jzxchiang It looks like the inter-canister message size limit is 2 MB.

Yes, I meant that one.

1 Like

For the text field you could serialize to binary by writing the length as one word followed by the text as Blob. For example define a function

import S "mo:base/ExperimentalStableMemory";
import Nat32 "mo:base/Nat32";
import Text "mo:base/Text";

    func storeText(offset : Nat32, value : Text) : Nat32 {
        let b : Blob = Text.encodeUtf8(value);
        let len : Nat32 = Nat32.fromIntWrap(b.size());
        S.storeNat32(offset, len);
        S.storeBlob(offset+4, b);
        4 + len
    };

and then call it like this:

offset += storeText(offset, street1);
offset += storeText(offset, street2);
offset += storeText(offset, city);
...
2 Likes

This example runs in the playground at the moment. However, I think the interface for ExperimentalStableMemory has recently changed to 64 bit address space so you may have to change all Nat32 to Nat64, depending on your environment.

1 Like

Thank you @timo. That was very nice of you to provide a code sample. It really cleared up my confusion.

Happy New Year!