Backup/Restore Function for a Canister

Developers
3

Hi Ashley,

We have two approaches here at Distrikt and the first one is kinda universal if you already use the linear Stable Memory (SM) during your upgrades, and I assume you do, otherwise you’re doomed :sweat_smile:

We are using Rust here, so I have no idea if you have the same access to the stable memory.

We take a snapshot of the state of the canister by serializing a data structure that is behind a RefCell that we will called state_store in our example here. To do so we call the pre_upgrade hook.
We then stream the data, with query calls, off chain by reading directly on the linear stable memory using ic_cdk::api::stable::stable64_read.

For restore, you then just have to do the reverse operation using ic_cdk::api::stable::stable64_write and use assign the deserialized data to your state_store global variable.

There is a few tricks here :

  1. Do not forget to use ic_cdk::api::stable::stable64_grow in order to grow the stable memory enough to dump the data into it.
  2. We use MessagePack as Serializer instead of the one provided by the IC. Because we need to know the exact size of the serialized data on the stable memory. Info that is currently unavailable by any means from the IC alone at the moment.
  3. At some point if there is too much data to be processed in a single update call when you’re calling the pre_upgrade hook (to save the sate to SM) or post_upgrade hook (to load the state from SM) directly it will fail because of the cycles limitation of the IC. A workaround is to use the limitation increase of the upgrade process in order to take advantage of the 40x bonus.
    Basically you force a canister upgrade (using dummy code) to dump the state to SM and then while the canister is up you stream the backup off chain. Same for restore : you disable the pre_upgrade hook, you stream the backup back to SM and finally you upgrade the canister. Disabling the pre_upgrade hook will keep the data on the SM safe and the post_upgrade hook will load back the canister state from your backup.

Please keep in mind that this method has been tested up to 2Gb of heap memory out of the 4Gb available on a canister. Using it on larger data will most likely result of an un-upgradable canister because the de/serialisation process, and so you will not be able to backup/restore.

I hope this is comprehensive enough and that it will help the IC community.

Cheers.

5 Likes