Verifying Key Resharing Proposals

Governance
1

As previously announced in this post, DFINITY will propose to temporarily pause and recover subnet uzr34 (which hosts the Internet Identity canister) on June 26 at 8AM UTC as part of the key resharing process for deploying the production vetKD key. This pause is expected to last 5–10 minutes.

In order to make this process verifiable end-to-end, the latest IC-OS version release contains a new tool to verify CUPs of halted subnets. This enables anyone to independently verify that the subnet:

  1. Was intentionally halted at a given CUP
  2. Was later restarted without any changes to its state

What is a CUP?

A Catch-Up Package (CUP) is a cryptographically signed snapshot of a subnet’s certified state. It is emitted periodically (e.g. every 500 blocks) and includes:

  • A threshold signature verifiable by the subnet’s public key
  • The block height of the CUP
  • A timestamp of when the CUP was created
  • A state hash representing the canonical state of the subnet at that point
  • The registry version under which the CUP was created

When a subnet is paused via governance proposal (as done previously in this proposal), the CUP created at the halting height represents the final checkpoint of this subnet. Subsequently, the halted subnet can only be restarted by adopting a new recovery CUP (as previously done here).

In order to prevent any (un)intentional changes to the state during this process, the state hash of the proposed recovery CUP should therefore be identical to the one referenced by the latest subnet CUP at the halting height.

What Does the Tool Do?

The new command-line tool offers functionality to verify the latest CUP of a subnet that was manually halted at the CUP height (i.e. as part of a maintenance). By inspecting this CUP, it allows anyone to determine which parameters should be used as part of a recovery proposal to restart the halted subnet. If the subnet has already been restarted, the tool will verify that the correct parameters were used according to the latest CUP.

Specifically, given a CUP persisted in a local file, the tool performs the following steps:

  1. Signature & Integrity Verification: Ensures that the CUP was genuinely produced by the subnet using threshold cryptography.

  2. Inspect & Display the CUP’s Parameters:
    a. The time, height & registry version at which this CUP was created.
    b. The state hash referencing the state checkpoint of the subnet.

  3. Verify that the subnet was indeed manually halted at this CUP, according to its registry version. Otherwise, this CUP does not necessarily represent the latest state of the subnet at the time it was halted.

  4. Verify that the subnet was recovered correctly. Any subsequent recovery proposal should specify:
    a. A height and timestamp that are greater than the ones of the local CUP. This ensures forward progress.
    b. A state hash that is identical to the one in the local CUP. This ensures that no changes were made to the subnet state.

Note that for now, the tool can only be used to verify recovery proposals for subnets that were intentionally halted at the CUP height as part of a scheduled maintenance. It cannot be used to verify recovery proposals for subnets that stalled unexpectedly (i.e. due to a problem introduced by a new replica version), because in this case the latest CUP does not necessarily represent the latest state of the subnet at the time of the stall.

Try it Yourself

In order to prepare for the recovery of uzr34 on June 26, today we are publishing a CUP of subnet io67a (google drive link), that will allow you to test and familiarize yourself with the tool’s functionality. After downloading the CUP artifact, it may be verified by executing the tool as follows:

  1. Checkout the current IC-OS version 3564b37939f037ba4d051ada88251c13954597d2 of the subnet:
git clone https://github.com/dfinity/ic.git
cd ic
git checkout 3564b37939f037ba4d051ada88251c13954597d2
  1. Copy the downloaded CUP into the IC directory
cp /path/to/io67a_cup.pb ./io67a_cup.pb
  1. Enter the development container:
ci/container/container-run.sh
  1. Build and run the CUP verification tool:
bazel run //rs/cup_explorer:cup_explorer_bin -- verify-cup-of-halted-subnet --cup-path /ic/io67a_cup.pb

Note that for this test, subnet io67a was not halted at the provided CUP. Therefore, the output of the tool should include the following error message:

"Verification failed: Subnet wasn't instructed to halt on this CUP. Therefore, this CUP is NOT guaranteed to represent the latest state of the subnet!"

During a production maintenance, the output should instead include the following instructions:

"Confirmed that subnet uzr34 was halted on this CUP."
"This means that the CUP represents the latest state of the subnet while the subnet remains halted."
"The subnet may ONLY be restarted via a recovery proposal using the same state hash as listed above."

Next Steps

During the key resharing maintenance of uzr34 on June 26, DFINITY will similarly publish the latest CUP of subnet uzr34 here on the forum, after the proposal to halt the subnet at a CUP height was adopted. At that point, we encourage you to execute the tool to verify that no state changes are made as part of the subsequent recovery proposal to perform the key resharing.

We will continue to improve these processes, as we work towards our goal of making all subnet recoveries end-to-end verifiable.

In the meantime, feel free to leave feedback and discuss the tool!

15 Likes
2

Nice work! Thanks @eichhorl. It’s great to see progress on this since discussions last year.

It’ll be cool once you can get around to actioning point 2 mentioned above too.

I’ve tried out the tool against the test CUP and it works nicely. Output below → 

INFO: Running command line: bazel-bin/rs/cup_explorer/cup_explorer_bin verify-cup-of-halted-subnet --cup-path /ic/io67a_cup.pb
Using mainnet NNS public key.
NNS public key being used:
-----BEGIN PUBLIC KEY-----
MIGCMB0GDSsGAQQBgtx8BQMBAgEGDCsGAQQBgtx8BQMCAQNhAIFMDm7HH6tYOwi9
gTc8JVw8NxsuhIY8mKTx4It0I10U+12cDNVG2WhfkToMCyzFNBWDv0tDkuRn25bW
W5u0y3FxEvhHLg1aTRRQX/10hLASkQkcX4e5iINGP5gJGguqrg==
-----END PUBLIC KEY-----

Registry client created. Latest registry version: 51387

Creating crypto component...
Jun 20 15:17:58.829 INFO s:/n:/ic_crypto_internal_csp/vault Proceeding with an in-replica csp_vault, CryptoConfig: CryptoConfig { crypto_root: "/tmp/ic_crypto_yYMyp2", csp_vault_type: InReplica }
Getting registry value of key crypto_record_nrp3d-ucxdx-h7pyr-q3mwb-qp2ur-dwgjf-2vdj5-5blib-rjvqf-mvow6-rqe_1 at version 51387...
Getting registry value of key crypto_record_nrp3d-ucxdx-h7pyr-q3mwb-qp2ur-dwgjf-2vdj5-5blib-rjvqf-mvow6-rqe_4 at version 51387...
Getting registry value of key crypto_tls_cert_nrp3d-ucxdx-h7pyr-q3mwb-qp2ur-dwgjf-2vdj5-5blib-rjvqf-mvow6-rqe at version 51387...
Getting registry value of key crypto_record_nrp3d-ucxdx-h7pyr-q3mwb-qp2ur-dwgjf-2vdj5-5blib-rjvqf-mvow6-rqe_3 at version 51387...
Getting registry value of key crypto_record_nrp3d-ucxdx-h7pyr-q3mwb-qp2ur-dwgjf-2vdj5-5blib-rjvqf-mvow6-rqe_5 at version 51387...

Reading CUP file at "/ic/io67a_cup.pb"
CUP integrity verified!

Checking CUP signature for subnet io67a-2jmkw-zup3h-snbwi-g6a5n-rm5dn-b6png-lvdpl-nqnto-yih6l-gqe...
Getting registry value of key catch_up_package_contents_io67a-2jmkw-zup3h-snbwi-g6a5n-rm5dn-b6png-lvdpl-nqnto-yih6l-gqe at version 51153...
CUP signature verification successful!

Latest subnet state according to CUP:
                TIME: 1750081799471456224, (2025-06-16 13:49:59.471456224 UTC)
              HEIGHT: 171663500
                HASH: db94cbaff73a1c74e780ff63a30484284da1a9dd69e5374098438390bda5e053
    REGISTRY VERSION: 51153

Verifying that the subnet was halted on this CUP...
Getting registry value of key subnet_record_io67a-2jmkw-zup3h-snbwi-g6a5n-rm5dn-b6png-lvdpl-nqnto-yih6l-gqe at version 51153...

thread 'main' panicked at rs/cup_explorer/src/main.rs:80:17:
Verification failed: Subnet wasn't instructed to halt on this CUP. Therefore, this CUP is NOT guaranteed to represent the latest state of the subnet!
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace

Once again, thank you :folded_hands: :slightly_smiling_face:

The CO.DELTA team will be ready to make use of this in the upcoming key resharing proposal.

6 Likes
3

It crossed my mind today while using this tool that the verification process may be simpler if an endpoint is simply queried to retrieve the hash of the latest CUP that a subnet was halted at, along with the height and timestamp. Can I ask what the motivation is for operating on the CUP itself when performing verification? My expectation is that it ultimately just comes down to the hash (and metadata such as height and timestamp).

Perhaps there’s something I’m missing (it’s been a long day).

4

This is something that was considered, however in that case, this endpoint would have to be queried from multiple subnet nodes during verification, in order to ensure that the majority of the subnet holds the same data.

The CUP has the advantage of being cryptographically verifiable via the subnet’s public key. Another advantage is that the CUP remains verifiable even after the recovery has completed, whereas an endpoint is only verifiable as long as that endpoint is served by the replica.

2 Likes