Temporary Canister Cycles Balance Drop when upgrading a canister

Developers
1

TLDR: Why do canisters temporarily dump excess cycles during an upgrade?

I was testing out upgrading canisters with an intermediate amount of data on main net (200+ MB) and ran into the following error.

Canister installation failed with `Canister ael4e-2yaaa-aaaan-qakia-cai is out of cycles: requested 80_000_590_000 cycles but the available balance is 204_292_897_802 cycles and the freezing threshold 186_257_633_040 cycles

This makes sense, although the upgrade requesting 80 billion cycles does sound like overkill. So I went ahead an topped up the canister and completed the upgrade.

I then went forward and started upgrading additional canisters (with similar amounts of data) while simultaneously pinging their cycles balances, and noticed the following strange behavior.

Steps:

  1. Before upgrading canister A, A has cycles balance = 0.615 T cycles
  2. Start upgrade of canister A
  3. Immediately poll the cycles balance of canister A (during the upgrade). At some point during the upgrade, the canister will temporarily return a cycles balance of ~0.535T (roughly 80 billion cycles less, gone in an instant)
  4. Several seconds later, the upgrade has finished. Poll the cycles balance of canister A again. Canister A now shows a balance of 0.608T cycles.

During canister upgrades, why do canister cycles balances temporarily dump cycles before getting “topped” back up? Is this to hold an excess to protect canisters against failed upgrades?

Also, in all of the failed cases, I found that the upgrade was requesting 80_000_590_000 cycles. Is there something magic about 80 billion cycles as to why this number was chosen?

1 Like
2

@domwoe pinged the execution team. I may at least know the answer to this question:

I think it is the result of this line, combined with a cost of 0.4 cycles per instruction (src here, search for Ten Update Instructions Execution).

1 Like
3

This is expected. We always reserve the maximum amount that can be used during any message execution (not only upgrades) and at the end we refund what was not used. This is to ensure that the work done can be paid for always. So what you observe is exactly how things are supposed to work.

As for the number itself, @Severin already gave pointers. One nit: the 590_000 is also easy to explain if you consider the base cost for a message. The formula is essentially 590_000 + max_instructions_for_message * 4 / 10 which, for an upgrade, gives you the “magic” 80_000_590_000 if you plug in the numbers.

5 Likes