Investigating recently failed SNS upgrade proposals

Governance NNS Governance
,
1

TLDR; earlier today, Dragginz experienced an SNS upgrade issue. We believe the issue was triggered by old proposal data. We are continuing to investigate. We ask you to not upgrade SNSes before we know more.

Earlier today, the Dragginz team reported a few failing SNS upgrade proposals in the past hours (see Dragginz Proposals - ICP Dashboard). The NNS team looked into these failure cases. We identified three distinct failure reasons:

  1. Duplicate upgrade proposals that were submitted within a very short period of time from one another

    Proposal 131 is a duplicate of proposal 130, which is the last successful SNS upgrade proposal for Dragginz.

  2. Upgrade proposals that started executing while other upgrade proposals were executing.

    This applies to proposals 132 till 135.

  3. Upgrade proposals that started executing, but never completed due to the WASM version not getting upgraded to the expected version.

    This applies to proposals 136 till 139.

The first two failure reasons are expected, i.e. the current SNS upgrade design does not support concurrent upgrade proposals. (The general recommendation is submitting one such upgrade proposal, getting it voted through, and only then submitting the next upgrade proposal.)

The third failure reason, however, is most likely indicative of a bug in the post-upgrade hook of the latest published SNS Governance, which panicked due to some old proposal data in the Dragginz SNS. We are currently reproducing this issue on a local testing environment, and will come up with a solution shortly.

While the risk seems to be low for SNSs (other than Dragginz) to submit upgrade proposals right now, we ask the teams to avoid doing so until the proper solution is ready.

12 Likes
2

Update.

  • The NNS team has confirmed that some data migration code has trapped, causing the post-upgrade hook to fail in the Dragginz SNS Governance canister. Specifically, the settle_proposals_if_reward_rates_are_zero function called by canister_post_upgrade iterates through previously made proposals and checks a set of conditions (before actually migrating the data). One of these conditions is that the proposal must be eligible for rewards, which is not the case for some Dragginz proposals, e.g., Dragginz Proposal: 122 - ICP Dashboard. The failed check caused the function to trap, failing the canister upgrade.
  • The only other SNS potentially affected is Gold DAO (as it also has voting reward rate set to zero). The corresponding teams have been notified.
  • We will propose a fix for upgrading Dragginz and Gold DAO tomorrow. All other SNSs are already safe to upgrade.
7 Likes
3

Quick update. The solution for the SNS upgrades has been merged, but we ran out of time testing it last week, so the corresponding upgrades haven’t been proposed yet. We thus notified the affected teams (Dragginz and Gold DAO) to hold off sending new SNS upgrade proposals until further notice.

3 Likes
4

With the following proposals being adopted, the SNS upgrade issue has been resolved for Dragginz and Gold DAO, which are now upgradable again.

3 Likes