Motoko Serialization

ICP is the future and us luck ones who are in, especially everyone who paid less than $50 each are going to be in such a good position as 2022 is going to be the year that shorts cover, which will shoot us to $250 and form a new base to take us to $1000 per ICP in 2022 :+1:t2::+1:t2::+1:t2:

Hey @KennyBoyNYC. I’m honored that your first post was in my thread, even if it’s a bit off topic. I won’t speculate on price, but let’s just say that $ICP is my primary long-term investment. :wink:

If you’re not a web developer yet, now is a good time to learn. Full-stack = React + Motoko.

1 Like

Ah. I’m honored. I’m not a developer but I heard amazing things bout Motoko :+1:t2::+1:t2::+1:t2:

1 Like

Let’s take a step back. If you store a lot of data in your canister then for save upgrades (to guarantee that the upgrade will never fail due to exceeding the cycle limit per execution) you will likely need to store data directly in stable memory with the low-level interface.


For the primitive types there are serialization functions provided in the ExperimentalStableMemory package that write the value directly into stable memory. For your own composite types you would have to write those serialization functions yourself. But hopefully the data types for your essential data that you need for disaster recovery can be kept simple.

If you do use the stable memory in that way already then it makes sense to use the serialization already present in the stable memory for your backup solution. Note that the data being in stable memory is already a form of backup because stable memory survives upgrades and if an upgrade fails because the pre/post upgrade hooks trap then the stable memory cannot be corrupted. So you are left to protect yourself against cases where the upgrade succeeds but the new code is buggy so that it corrupts your data after the upgrade.

To back up the data in the stable memory I would just take the entire memory pages and ship them off as Blob to another canister. That way the type transmitted is always Blob and the backup canister can be completely agnostic to the content and the types used in the content (as you wanted). You should be able to send at least 2 MB in a single inter-canister message. With the stable memory being organized in memory pages of 64kB you can then ship 32 pages at once.


Since you mentioned Text. If you have Text you can convert it to Blob with Text.encodeUtf8 first and then write it to stable memory with storeBlob.

To back up the data in the stable memory I would just take the entire memory pages and ship them off as Blob to another canister.

To clarify, do you mean to use the loadBlob function?

You should be able to send at least 2 MB in a single inter-canister message. With the stable memory being organized in memory pages of 64kB you can then ship 32 pages at once.

Oh interesting, I thought the 2 MB message size limit only applied to ingress messages. I didn’t know they also applied to inter-canister messages.

Thank you @timo for your comprehensive explanation. This gives me a lot to think about.

Due to the experimental warning, I have intentionally left out ExperimentalStableMemory from the solution. However, I am very hopeful about leveraging this library in the future. I have a few questions on that topic if you have the time.

  1. It sounds like the only way to get a blob from an instance of a complex type is to custom serialize it to a string format like JSON and then call Text.encodeUtf8. Is my understanding correct?
  2. If I need to restore all pages of stable memory after an upgrade to some HashMaps, could I still run into the cycle limit?
  3. To echo @jzxchiang, does that 2 MB message size limit apply to inter-canister messages?

Certainly not the only way. What complex type are you thinking about? You can probably serialize it directly to binary.

Theoretically, yes. If that happens then you could spread building the HashMap over several messages. The upgrade could leave the canister in a state in which only a post-upgrade function can be called and no other functions. And then you can call the post-upgrade function multiple times and each time you call it a few more entries get added to the HashMap. There are also ways for a canister to call itself in a loop, then it would be automated.

As far as I know, yes. Inter-canister messages end up in blocks and there is a block size limit so there must be a message size limit, too.

Thanks again. I like your idea to copy the stable data with multiple operations after the upgrade is finished. I’ll do some testing for the message size limit since the IC code is a bit over my head.

If you could show me how to do that, it would solve my biggest problem. Here are some examples of what I would call complex types:

public type Address = {
   id: Nat32;
   street1: Text;
   street2: Text;
   city: Text;
   province: Text;
   postalCode: Text;
   country: Text;

public type Person = {
   id: Nat32;
   firstName: Text;
   lastName: Text;
   emails: List<Text>;
   addresses: List<Nat32>;

@timo @jzxchiang It looks like the inter-canister message size limit is 2 MB.

Yes, I meant that one.

1 Like

For the text field you could serialize to binary by writing the length as one word followed by the text as Blob. For example define a function

import S "mo:base/ExperimentalStableMemory";
import Nat32 "mo:base/Nat32";
import Text "mo:base/Text";

    func storeText(offset : Nat32, value : Text) : Nat32 {
        let b : Blob = Text.encodeUtf8(value);
        let len : Nat32 = Nat32.fromIntWrap(b.size());
        S.storeNat32(offset, len);
        S.storeBlob(offset+4, b);
        4 + len

and then call it like this:

offset += storeText(offset, street1);
offset += storeText(offset, street2);
offset += storeText(offset, city);

This example runs in the playground at the moment. However, I think the interface for ExperimentalStableMemory has recently changed to 64 bit address space so you may have to change all Nat32 to Nat64, depending on your environment.

1 Like

Thank you @timo. That was very nice of you to provide a code sample. It really cleared up my confusion.

Happy New Year!

This is a great thread and directly related to a problem I want to solve …

As @jzxchiang asked and @timo replied with:

I am trying to do just that … not sure if there are examples of that somewhere ?

and on the same subject, lets assume I can figure out how to create the Blob I want and can break it into 2MB chunks of pages… Can you turn an array of complex objects into a Blob relatively easily? … if I were to take the example from @Motokoder, i.e. Address, and had an array of addresses i.e.:

var theAddresses : [Address] = [] ;

and then I wanted to turn “theAddresses” into a Blob so I can chunk it up and send to another canister …

I feel like I am 80% to the finish line, but still a little confused … any assistance would be great.
Thanks in advance … and I know this thread is a little old, but like I said, right on target with what I am trying to do …

1 Like

Check out the to_candid and from_candid methods introduced in the links provided in this post


Holy guacamole … seems to be exactly what I am looking for … hard to find, but for the next person along the same path … here is a direct to the doc on it :slight_smile:

(fyi - the forum strips the #candid-serialization … search for “Candid Serialization” in this page … )

Muchas gracias @icme (a.k.a. Canscale) … and again thanks to @jzxchiang @timo :upside_down_face:

going to try it right now …


Also found this helpful as a reference …

thanks @Gabriel :slight_smile:


and thanks to @matthewhammer and @claudio for building it in the first place (fyi … need to upgrade to 0.10.1 to get the Motoko version with the methods … otherwise will get “unbound variable to_candid” ) … rocking on …


As a closer to this subject … the from_candid method returns an optional type or () (?Type) … this was a new thing for me, as it will not compile if you take that value and try to use it … so we need a switch to check whether it worked … even though it will assert if the “from_candid” fails … here is a quick code example that may prove useful to the next Explorer walking this path:

var tempBlobReturned : ?ArchiveType = from_candid(tempBlob); 

switch tempBlobReturned {
  case (?val) {
    Debug.print("RESTORE SUCCESS - have a value" ) ;
    tempArchiveRestored:=val ;
  case null {
    Debug.print("RESTORE FAILED - have no value " ) ;
};// end switch

Happy to respond if anything does not make sense here … extrapolated this solution from the following post:

needless to say, but very obtuse … thanks @rossberg
please rock on …

1 Like