Canister backup

It isn’t the same in cycle costs though. A one time fork call would not require pushing all the bits through inter canister calls which each have a per byte and ingress charge. 2000*those fees is a good number of cycles.

There is no public price for that, so I won’t argue.
But since it is the same load, I would speculate that it should cost the same amount of money for nodes.

Thanks @jzxchiang , that’s exactly what I would like to do !

But I don’t know where to begin with to implement it in a canister…
Could you please provide some kind of short code example in Rust of how to access and read the wasm linear memory ? I would forever be grateful :bowing_man:

I might buy you a beer if you come at the European Blockchain Convention in June :call_me_hand:

To be more specific, I have a struct that lives inside a RefCell that holds all the data that I want to back up off chain, so my best wish is to stream it through queries chunk by chunk.

So my guess on tackling this issue are :

  1. Find a way to access the wasm linear memory and find the boundaries of my struct in order to chunk and retrieve. My preferred solution as it would imply no extra costly memory allocation on a already resource constrained environment.
  2. Query page by page from stable memory as pointed out earlier, but it would mean data duplication :confused:
  3. Serializing my struct and then retrieve it chunk by chunk. Again, it would imply data duplication, which I’m trying hard to avoid as it could outgrow the memory limitation of the canister at some no so distant point in the future.

I would be most grateful if someone could give me some insight on this matter :pray:

Hey…If your data is public, you could stream it using the streaming call back and then just curl your state. If it needed to be secret you could add a secret token to the query string. In fact I’m goin to do this now that I’ve thought about it

FWIW I’ve extended my fork of quill to support any ingress messages with raw I/O. That means you can use it to read and write stable memory pages as bytes bypassing the pretty-printing by dfx and hence read/load up to 2Mb at once.

Here is an example how to read data:

./qu --pem-file <..> raw <canister_id> <method> --args "<candid_encoded>" --query | ./qu send --yes --raw - > ./data.bin"

To load the file back:

./qu --pem-file <..> raw <canister_id> <method> --args-file ./data.bin | IC_URL=http://127.0.0.1:8000 ./qu send --yes -

(Note that the loading assumes loading to your local replica, that’s why it has the IC_URL set to localhost and that how I suggest to test the backup, obviously)

This can now be easily wrapped with a bash script to read the memory page by page and load it back page by page. The next step would be to get rid of Candid serialization altogether, but I’m still working on a patch for Rust CDK for that.

3 Likes

Hey, I shared one of our technique here at Distrikt.app in an other post if anybody is still interested.

2 Likes

Any news on native canister backups?

1 Like

Hi! There is a community discussion thread on backup and restore Canister backup and restore [Community Consideration]. We would like to hear your thoughts there.