Canister Migration

Developers
1

Introduction

We are excited to announce that DFINITY started working on enabling canister migration from one subnet to another subnet. This feature allows canister developers to move their canisters away from a busy subnet or onto a subnet with desired subnet features (e.g., the European subnet) or replication (e.g., the fiduciary subnet).

Canister migration preserves canister state, cycles, and canister ID. On the other hand, canister migration does not preserve transient state (metrics, logs, history) and reserved cycles. Last but not least, the canister must be stopped (no open call contexts) before canister migration.

In the remainder of this post, we propose a flow for canister migration and would like to kindly ask you for your feedback on our proposal.

High-Level Overview

Canister migration is split into three phases:

  • Canister State Migration: via downloading and uploading a canister snapshot;
  • Canister Cycles Migration: via cycles wallet;
  • Canister ID Migration: via a dedicated NNS canister.

Canister State Migration

Canister state migration proceeds via downloading a canister snapshot of the original canister and uploading that canister snapshot to a new canister on the target subnet (with a temporarily different canister ID). This flow will be enabled by extending the management canister API with endpoints to:

  • read snapshot’s metadata;
  • read snapshot’s paginated binary data (wasm module, main, stable memory);
  • read snapshot’s wasm chunk store (by chunk hash);
  • create new snapshot from metadata (instead of the current canister state);
  • write snapshot’s paginated binary data (wasm module, main, stable memory);
  • write snapshot’s wasm chunk store (chunk-wise).

The new management canister API will be used by:

  • a pair of new dfx commands to download and upload canister snapshots stored in a directory on disk (off-chain) and
  • by a “snapshot migration canister” (developed by DFINITY) providing an endpoint to migrate a canister snapshot to another canister (on-chain).

The on-chain flow using a snapshot migration canister allows for independent data integrity verification (using reproducible builds, canister history, and logs stored in the snapshot migration canister).

We point out that downloading and uploading canister snapshots is useful beyond canister migration: it enables keeping multiple (backup) snapshots (ICP currently only allows one snapshot per canister), release/local testing with mainnet/preconfigured state, and disaster recovery.

Canister Cycles Migration

Canister cycles migration can be performed by deploying the cycles wallet to the original canister after taking its snapshot and transferring cycles to the target canister using that cycles wallet. The individual steps are already available as dedicated dfx commands (dfx identity deploy-wallet and dfx canister deposit-cycles).

Canister ID Migration

After canister state migration, there are two canisters: the original canister with the original canister ID on the original subnet and the new canister with a temporarily different canister ID on the target subnet that has the same state as the original canister. The goal of canister ID migration is to rename the new canister on the target subnet to the original canister ID and to get rid of the original canister on the original subnet. A principal controlling both canisters triggers canister ID migration by

  • adding a new dedicated NNS canister performing canister ID migration as an additional controller of both canisters; and
  • making a call to the NNS canister performing canister ID migration.

A new dedicated dfx command will encapsulate these steps. Once the call to the NNS canister successfully completes, the new canister on the target subnet has the original canister ID and controllers. Otherwise, canister migration can be aborted by restoring the original canister from its snapshot.

Discussion

We are happy to receive feedback or answer questions here, and we’ll share updates on the progress in this forum thread.

13 Likes
2

I think the most tricky part is the Canister ID migration.

Could you please elaborate a bit more in that?

This will need changes at protocol level no? It can’t be done at “canister” level right?

It’s unclear to me how it “swaps” one for the other. How easy / fast it would propagate that change across the boundary nodes, etc.

Since it’s a temporary ID and then a swap in two subnets. Won’t somewhere be the risk of having the same Canister ID “alive”, even if temporary, in two subnets?

Thanks :pray:

2 Likes
3

Indeed. Canister ID migration will be supported by the core protocol using the management canister abstraction: there will be a new endpoint on the management canister (implemented directly by replicas) that will be called by a new NNS canister (ensuring all requirements for a canister ID migration are met).

Technically, the canister ID migration would proceed as follows:

  • the NNS canister sets itself as the only controller of both canisters (so that no other principal can interfer during canister ID migration);
  • the NNS canister performs checks ensuring that all requirements for a canister ID migration are met (e.g., both canisters are stopped);
  • the NNS canister calls the management canister (replicas) on the target subnet to (atomically) rename the target canister;
  • [at this point there are two canisters with the same canister ID, but they are both stopped and only controlled by an NNS canister]
  • the NNS canister updates the routing table in the ICP registry;
  • the NNS canister waits until the registry changes propagates to the two subnets hosting the two canisters;
  • the NNS canister deletes the original canister;
  • the NNS canister restores the controllers of the original canister to be the controllers of the target canister (with the original canister ID by now).

The changes propagates just like any registry change through the IC (i.e., its replicas/subnets and boundary nodes).

This risk is mitigated by the canisters being stopped only controlled by the NNS canister (so that nobody else can restart them) during canister ID migration.

4 Likes
4

Yes, now I understand. This is pretty cool :sunglasses:

Ok, now I see how that would work, thank you for the detail. :pray:

And congrats, this wasn’t an easy problem to crack. I really like the NNS canister being the sole controller and processor of all these steps. :+1::clap:

1 Like
5

This step was previously not agreeable based on discussions with other members of Dfinity engineering based on the reasoning that this would lead to a fragmented routing table (hence reducing the benefits of a contiguous range of canister IDs in the routing table?).

How has this changed now and how would this affect canister lookup speeds on all subnets?

@mraszyk

3 Likes
6

tl;dr The performance of canister lookups in the routing table will be addressed by refactoring the routing table in the registry and certified state tree.

Currently, the entire routing table (across all subnets) is stored as a single registry value and the entire collection of canister ranges of a subnet is storead as single value in the certified state tree. As part of the canister migration feature, we plan to split the routing table into multiple values both in the registry and the certified state tree to facilitate efficient canister lookups.

2 Likes
7

Will it also be possible to duplicate a canister?

You mentioned it above that there will temporarily be two canisters on the two subnets. Eventually one will get removed. Can I keep both if I wanted to?

Or maybe this question is not specific to migration? Because I could ask the same for creating a second copy based on a snapshot on the same subnet.

1 Like
8

Indeed. The extended canister snapshot API will also allow to “duplicate” or “clone” a canister.

Absolutely, you can keep both. Of course, this only works if your canister doesn’t depend on any particular canister ID or can be configured to work with a different canister ID.

Indeed. “Duplicating” a canister using canister snapshots is also possible on the same subnet (whereas canister ID migration only makes sense if two different subnets are involved).

1 Like
9

Let me guess, from the perspective of somebody that is holding people’s applications hostage, you would NOT want this change to be implemented?

Anybody have a feeling what saikat’s views are on this?

10

Hi Adam :waving_hand:

I’m ALWAYS HAPPY to answer any questions you have of us :slight_smile:

FAR from it. We’ve been eagerly waiting for it

You can see that we’ve been asking for canister migrations as early as September 2022.

Here’s us asking about it when it was still being called “Canister forking”

And then PLEADING for it as recently as October 2024

Hope that was insightful :blush:

11

Would this mean I can keep two or more canisters that have the same ID and use that as a way to scale?

12

No, there can be only one running canister with the same canister ID at any time. What I meant is that the state of an existing canister can be “duplicated” or “cloned” easily via the protocol, i.e., without the canister WASM exposing custom endpoints to download and upload its state.

2 Likes
13

Hello!

Is it possible to add permanent stable memory replication?

If I want to have a canister data replica in another subnet, I will be able to make a canister clone. But to replicate new changes added after the data cloning I will need another cloning.
Q1:
Will be the next cloning cheaper and faster? Or it will not use some kind of incremental cloning?

I understand that it could be made by inter-canister calls, on each data update.
Q2:
But maybe there could be some built-in method for stable-memory replication that could be faster and cheaper than inter-canister call on each data update?

14

Why would you like to do that?

It won’t be cheaper/faster because the protocol won’t keep track and expose the stable memory regions touched since a particular snapshot.

That could be a follow up feature (no promise at this point).

15

Why would you like to do that?

I’m thinking about high-load application, where we can loadbalance some calls.

Moreover I’m thinkng about sharding canister with a lot of data. When there is too lot of data for canister, we would like to split it (in two subnets). And we will need to split the data. But this operation will take long time, so we can have the already prepared replica, to split it fast.