Proposal: Making Variant Fields Optional for Schema Evolution

Developers
1

Hi everyone,

We’d like to share a proposal to update certain data structures in the management canister interface to make them more future-proof.

Currently, some records are defined in ways that limit their ability to evolve. For example:

type change = record { 
   timestamp_nanos : nat64;
   canister_version : nat64;
   origin : change_origin;
   details : change_details;
};

with change_details defined as a variant type:

type change_details = variant {
    creation : record {
        controllers : vec principal;
        environment_variables_hash : opt blob; 
    };
    code_uninstall;
    code_deployment : record {
        mode : variant { install; reinstall; upgrade };
        module_hash : blob;
    };
    load_snapshot : record {
        canister_version : nat64;
        snapshot_id : snapshot_id;
        taken_at_timestamp : nat64;
        source : variant {
            taken_from_canister : reserved;
            metadata_upload : reserved;
        };
    };
    controllers_change : record {
        controllers : vec principal;
    };
};

The problem

Candid variant types are closed enums. Once you define a set of branches, you cannot safely:

  • Rename branches

  • Add new branches without creating a breaking change

In Candid, new variant branches are not ignored. Decoding using the legacy type fails upon encountering an unknown branch in the new type. That means evolving a type like change_details is not possible without possibly breaking existing canisters.

The limitation does not just affect change_details. Other types that use variants—such as source and globals in ReadCanisterSnapshotMetadataResponse—face the same issue. Both are variant-based and therefore cannot evolve safely in the current form. For the same reasons as with change_details, these fields also need to become optional so that they can be extended in the future without introducing breaking changes.

Proposed solution

To preserve flexibility and forward compatibility, we propose making the fields optional:

type change = record {
    timestamp_nanos : nat64;
    canister_version : nat64;
    origin : change_origin;
    details : opt change_details;
};

Similarly, in ReadCanisterSnapshotMetadataResponse, the source and globals fields will also be migrated to optional fields:

type read_canister_snapshot_metadata_response = record {
    source : opt variant { /* ... */ };
    globals : vec opt variant { /* ... */ };
    // ... other fields remain unchanged ...
};

This way:

  • Future changes can introduce new variant branches without breaking existing consumers.

  • We give ourselves the ability to evolve the schema gradually, rather than being locked into today’s definition.

Why this matters

Core management canister records are long-lived pieces of state that must remain stable over time. As features evolve (e.g., new settings, new operations, or extended snapshot metadata), these records will need to be extended. Without this adjustment, each such change risks breaking compatibility with existing tooling and canisters.

By making these fields optional, we ensure long-term stability while still allowing future extensibility.

7 Likes
3

Is there a reason you cannot put this at an endpoint with _v2(or alternative) at the end?

If people are adding new functionality, they are, by definition, needing to upgrade canisters, and this would force them to migrate to the new _v2 function to use the new variants.

I know I have deployed code that uses the snapshot functionality. I’m guessing this change would break it.

1 Like
4

I appreciate your input. We considered introducing a _v2 endpoint, but decided it’s better to update the existing API for a few reasons:

  • Introducing multiple versions of the same endpoint would fragment the API. This makes maintenance harder and increases the risk of inconsistencies between versions.

  • In the case of snapshot metadata, outdated endpoints would eventually have to return an error if the new metadata cannot be encoded in the legacy response type.

  • The change_details record is tightly coupled with settings changes. Using a single endpoint ensures that all changes are reflected explicitly in the canister’s history. With multiple versions, users might miss information if they use an outdated endpoint, it might not be so immediately obvious and most likely confusing as well. With optional fields, users must actively and explicitly handle null values, which is safer.

5

Ok. Degrade away I guess.

image
image721×185 13.4 KB

a man in a jacksonville jaguars shirt stands in a crowd

You caught me on a chippy morning. This stuff isn’t supposed to break. It just creates mountains of work and testing when it does, and just absolutely destroys credibility and trust that you can build going forward. I get that someone screwed up as null variant has been the best practice since genesis, even when it looks odd and didn’t seem to make sense…and I know from experience that it is difficult to stay disciplined with it, but when breaking it to fix the glitch doesn’t seem very immutable.

1 Like