Objective
To ensure canister upgrade safety, dfx needs to verify that an upgrade can proceed without:
- breaking clients (due to a Candid interface change). We specifically designed Candid to define, in a checkable manner, what a safe upgrade is. This is expressed in terms of subtyping rules on Candid interfaces. Joachim recently explained these in a blog series. We are actually able to formally prove, as a form of machine-verified soundness statement, that these rules correctly prevent clients from breaking.
- discarding Motoko stable state (due to a change in stable declarations). While stable variables are not part of the public interface of a canister, they are a sort of private interface between two versions of a canister, and generally need checking in an analogous way. Concretely, the new canister should still be able to read the contents of existing stable variables, which requires restricting changes to their type across an upgrade, otherwise a canister might accidentally lose data.
Compiler support
PR #2887 adds support for manually checking compatibility of Candid and stable signatures using moc.
Roughly,
moc --idl foo.mo will write the Candid interface of an actor(class) to file foo.did.
You can use tool didc to check that the dids are compatible
didc --check new.did old.did
The Candid interfaces can evolve to a Candid subtype (note the inversion of the relation), so that old clients can use the new interface without breaking.
moc --stable-types foo.mo will write the stable signature of an actor(class) to file foo.most.
moc --stable-compatible old.most new.most will check that the stable interface can evolve from old.most to new.most in a type safe way without unintentional data loss.
PR #2897 has some examples to explain what is a compatible signature change and what isn’t.
The basic rules for stable-compatible are that you can evolve
- a stable variable to a super-type with the same mutability (so that the upgrade can consume the old value)
- add a new stable variable with different name and any type (it takes its value from the initializer when first introduced)
In practice, with the current implementation, one can safely change the mutability of a stable variable in an upgrade and drop an existing stable variable, but we have decided to error on those cases for now (but might only warn in future).
Besides writing the Candid interface and stable signature as files, the Motoko compiler also compiles these interfaces into the Wasm module as custom sections.
- Custom section “candid:service” stores the interface for the running (initialized) canister, which removes the initialization arguments.
- Custom section “candid:args” stores the initialization arguments. The argument types can refer to types defined in the candid:service custom section.
- Custom section “motoko:stable-types” stores the signatures for stable variables.
The compiler flag --public-metadata decides if the custom section can be retrieved publicly, or only by the controllers of the canister.
Interface spec change
For dfx to perform the compatibility check, we need to be able to download the Candid interface and the stable signatures from the running canister. This requires a general solution to expose canister metadata from the replica, preferably in a certified manner.
We add a new path in the state tree to expose canister metadata. Specifically, /canister/<canister_id>/metadata/<name> contains the content of the Wasm custom section named “icp:public ” or “icp:private ”.
The prefix “icp:public ” means the metadata can be accessed publicly, and the prefix “icp:private ” means the metadata is only accessible by the controllers of the canister.
As the specification repo of the Internet Computer is not open source yet, please see the proposed diff here.
The benefit of exposing metadata from the state tree is that it can be accessed from the HTTP endpoint, and the result is certified for free. The downside is that there is no programmatic way of accessing the state tree within the canister. So it is currently not possible to check for compatibility of a canister, if all of the controllers are canisters, not user principals. There is an ongoing RFC to consolidate the metadata interface, so that both canisters and users can access the metadata in a uniform way.
dfx integration
PR #1926 checks for upgrade compatibility in dfx canister install, and allows users to override the warning if they understand the consequence.
