Motoko Serialization

[jzxchiang]

To back up the data in the stable memory I would just take the entire memory pages and ship them off as Blob to another canister.

To clarify, do you mean to use the loadBlob function?

You should be able to send at least 2 MB in a single inter-canister message. With the stable memory being organized in memory pages of 64kB you can then ship 32 pages at once.

Oh interesting, I thought the 2 MB message size limit only applied to ingress messages. I didn’t know they also applied to inter-canister messages.

[Motokoder]

Thank you @timo for your comprehensive explanation. This gives me a lot to think about.

Due to the experimental warning, I have intentionally left out ExperimentalStableMemory from the solution. However, I am very hopeful about leveraging this library in the future. I have a few questions on that topic if you have the time.

1. It sounds like the only way to get a blob from an instance of a complex type is to custom serialize it to a string format like JSON and then call Text.encodeUtf8. Is my understanding correct?
2. If I need to restore all pages of stable memory after an upgrade to some HashMaps, could I still run into the cycle limit?
3. To echo @jzxchiang, does that 2 MB message size limit apply to inter-canister messages?

[timo]

@Motokoder

It sounds like the only way to get a blob from an instance of a complex type is to custom serialize it to a string format like JSON and then call Text.encodeUtf8.

Certainly not the only way. What complex type are you thinking about? You can probably serialize it directly to binary.

@Motokoder

If I need to restore all pages of stable memory after an upgrade to some HashMaps, could I still run into the cycle limit?

Theoretically, yes. If that happens then you could spread building the HashMap over several messages. The upgrade could leave the canister in a state in which only a post-upgrade function can be called and no other functions. And then you can call the post-upgrade function multiple times and each time you call it a few more entries get added to the HashMap. There are also ways for a canister to call itself in a loop, then it would be automated.

@Motokoder

does that 2 MB message size limit apply to inter-canister messages?

As far as I know, yes. Inter-canister messages end up in blocks and there is a block size limit so there must be a message size limit, too.

[Motokoder]

Thanks again. I like your idea to copy the stable data with multiple operations after the upgrade is finished. I’ll do some testing for the message size limit since the IC code is a bit over my head.

@timo

Certainly not the only way. What complex type are you thinking about? You can probably serialize it directly to binary.

If you could show me how to do that, it would solve my biggest problem. Here are some examples of what I would call complex types:

public type Address = {
   id: Nat32;
   street1: Text;
   street2: Text;
   city: Text;
   province: Text;
   postalCode: Text;
   country: Text;
};

public type Person = {
   id: Nat32;
   firstName: Text;
   lastName: Text;
   emails: List<Text>;
   addresses: List<Nat32>;
};

[Motokoder]

@timo @jzxchiang It looks like the inter-canister message size limit is 2 MB.

[timo]

@jzxchiang

do you mean to use the loadBlob function

Yes, I meant that one.

[timo]

@Motokoder

Here are some examples of what I would call complex types:

public type Address = {
   id: Nat32;
   street1: Text;
   street2: Text;
   city: Text;
   province: Text;
   postalCode: Text;
   country: Text;
};

public type Person = {
   id: Nat32;
   firstName: Text;
   lastName: Text;
   emails: List<Text>;
   addresses: List<Nat32>;
};

For the text field you could serialize to binary by writing the length as one word followed by the text as Blob. For example define a function

import S "mo:base/ExperimentalStableMemory";
import Nat32 "mo:base/Nat32";
import Text "mo:base/Text";

    func storeText(offset : Nat32, value : Text) : Nat32 {
        let b : Blob = Text.encodeUtf8(value);
        let len : Nat32 = Nat32.fromIntWrap(b.size());
        S.storeNat32(offset, len);
        S.storeBlob(offset+4, b);
        4 + len
    };

and then call it like this:

offset += storeText(offset, street1);
offset += storeText(offset, street2);
offset += storeText(offset, city);
...

[timo]

This example runs in the playground at the moment. However, I think the interface for ExperimentalStableMemory has recently changed to 64 bit address space so you may have to change all Nat32 to Nat64, depending on your environment.

[Motokoder]

Thank you @timo. That was very nice of you to provide a code sample. It really cleared up my confusion.

Happy New Year!

[mbeekman]

This is a great thread and directly related to a problem I want to solve …

As @jzxchiang asked and @timo replied with:

@timo

To back up the data in the stable memory I would just take the entire memory pages and ship them off as Blob to another canister. That way the type transmitted is always Blob and the backup canister can be completely agnostic to the content and the types used in the content (as you wanted). You should be able to send at least 2 MB in a single inter-canister message. With the stable memory being organized in memory pages of 64kB you can then ship 32 pages at once.

I am trying to do just that … not sure if there are examples of that somewhere ?

and on the same subject, lets assume I can figure out how to create the Blob I want and can break it into 2MB chunks of pages… Can you turn an array of complex objects into a Blob relatively easily? … if I were to take the example from @Motokoder, i.e. Address, and had an array of addresses i.e.:

var theAddresses : [Address] = [] ;

and then I wanted to turn “theAddresses” into a Blob so I can chunk it up and send to another canister …

I feel like I am 80% to the finish line, but still a little confused … any assistance would be great.
Thanks in advance … and I know this thread is a little old, but like I said, right on target with what I am trying to do …

[icme]

Check out the to_candid and from_candid methods introduced in the links provided in this post

[mbeekman]

Holy guacamole … seems to be exactly what I am looking for … hard to find, but for the next person along the same path … here is a direct to the doc on it

Language quick reference | Internet Computer Home

(fyi - the forum strips the #candid-serialization … search for “Candid Serialization” in this page … )

Muchas gracias @icme (a.k.a. Canscale) … and again thanks to @jzxchiang @timo

going to try it right now …

[mbeekman]

Also found this helpful as a reference …

Candid Serialisation: to_candid and from_candid

In the light of the new candid serialisation I thought I’d share a simple example to serialise/deserialise a type. The debug will print: [Canister rrkah-fqaaa-aaaaa-aaaaq-cai] ?{num = 1; text = "test"} Hope this help others.

thanks @Gabriel

[mbeekman]

and thanks to @matthewhammer and @claudio for building it in the first place (fyi … need to upgrade to 0.10.1 to get the Motoko version with the methods … otherwise will get “unbound variable to_candid” ) … rocking on …

[mbeekman]

As a closer to this subject … the from_candid method returns an optional type or () (?Type) … this was a new thing for me, as it will not compile if you take that value and try to use it … so we need a switch to check whether it worked … even though it will assert if the “from_candid” fails … here is a quick code example that may prove useful to the next Explorer walking this path:

var tempBlobReturned : ?ArchiveType = from_candid(tempBlob); 

switch tempBlobReturned {
  case (?val) {
    Debug.print("RESTORE SUCCESS - have a value" ) ;
    tempArchiveRestored:=val ;
  };
  case null {
    Debug.print("RESTORE FAILED - have no value " ) ;
  };
};// end switch

Happy to respond if anything does not make sense here … extrapolated this solution from the following post:

Сannot producer expected type

You need to switch and pattern-match on the optional value: switch result { case (?val) { .../* val : Text here*/... }; case null { ... }; } If you have multiple of such tests, take a look at the do? construct, which comes in handy and e.g. allows collapsing multiple switches into one: let result = do? { List.last(list1)! + List.last(list2)! + 1 }; switch result { case (?n) { ... handle result ... } case null { ... handle error ... } } The ! operator inside a do? block converts ?T t…

needless to say, but very obtuse … thanks @rossberg …
please rock on …