Cannot delete canisters anymore

[peterparker]

Has anyone else experiencing issues recently trying to delete canisters?

I am not able to delete canister anymore, I get following error when I try to do so:

Error: Call was rejected:
Request ID: 6c6d2f761b741…9866dcd61e900cb
Reject code: 4
Reject text: IC0501: Canister jjmme-aqaaa-aaaai-ab5ya-cai is out of cycles: requested 893853709998 cycles but the available balance is 893857809998 cycles and the freezing threshold 2057103111 cycles

I did not change my code nor faced the error previously.

Not later than a week ago I even used the function multiple times to determine what are the minimal cycles to retain upon deleting a canister (see post). Therefore quite surprise it does not work anymore out of the blue.

I also get the error when I create a fresh canister (create → do nothing else → delete → error).

So anyone having issues? Any idea what’s the issue?

My delete function:

let deckBucket = actor(Principal.toText(canisterId)): actor { transferCycles: () -> async () };

await deckBucket.transferCycles();

await ic.stop_canister({ canister_id = canisterId });

await ic.delete_canister({ canister_id = canisterId });

The transfer cycles:

public func transferCycles(caller: Principal): async () {
            let balance: Nat = Cycles.balance();

            let cycles: Nat = balance - 4_100_000;

            if (cycles > 0) {
                Cycles.add(cycles);
                await ic.deposit_cycles({ canister_id = caller });
            };
        };

[peterparker]

Alright I can reproduce the exact same issue on mainnet with a sample repo.

To deploy my canister on mainnet, I create a canister in nns-dapp, link my controller, add a canister_ids.json to the project and then deploy with the command dfx deploy --network=ic --no-wallet

Can it be the root cause of the problem or there is indeed an issue elsewhere?

Note: it works just fine on the local simulated network - it does not on mainnet

[claudio]

Thanks for the report and repro!

Can you confirm that the version that worked recently and the failing version were compiled with the same moc or dfx version? That’s to rule out a bug in Motoko.

I’ve also raised the issue internally with the execution team in case they can explain it.

[peterparker]

Thanks Claudio.

Cannot be 100% sure but it might have stopped to work after I upgraded dfx indeed. I used to used v0.8.4 if I remember correctly.

[claudio]

An internal user reports:

IIRC, I once ran into a similar error when I didn’t upgrade my wallet after updating dfx.
I just created an account on the forum and can’t reply yet. Can you them if they have upgraded their wallet? dfx wallet upgrade

[peterparker]

❯ dfx wallet --network ic upgrade
Error: Could not find wallet for “default” on “ic” network.

I create the canister in nns-dapp, link my controller and then deploy with --no-wallet therefore I guess there isn’t a wallet locally?

[Severin]

@claudio

internal user

That’s me.

--no-wallet (which isn’t needed after the latest dfx upgrade anymore, it’s implicit now) means that the wallet is not used to sign/send/? the commands. Your identity’s principal is the origin of your request.
It is still necessary to have a wallet to withdraw the cycles (unless you want to use --no-withdrawal and lose the cycles).
According to the error message you get, it looks like you don’t have a cycles wallet configured with your identity ‘default’. If you already have a cycles wallet, you can set it with dfx identity set-wallet <wallet-id> so that any command you perform with the selected identity will use that wallet to take cycles from/put cycles into. If you do not have one already, go to Smart Contracts on the Internet Computer and follow 4.4.2.
If you don’t want to link the cycles wallet by default, look at the output of dfx canister delete --help , especially the --withdraw-cycles-to-canister flag

[peterparker]

@Severin

If you already have a cycles wallet, you can set it with dfx identity set-wallet <wallet-id>

Thanks for the answer. I create the canisters with nns-dapp and transfer the ICP to the canisters’ cycles as well - i.e. I don’t use dfx to create canister or handle cycles, I do this in nns-dapp.

My local identity is added as controllers of the canisters.

Therefore not sure chapter 4.4.2 of the documentation apply to this case?

[ulan]

For each cycle transfer IC checks that the remaining cycles are above the freezing threshold. The freezing threshold depends on the memory usage and compute allocation of the canister, so it can increase over time.

The transferCycles function hardcodes 4_100_000 as the freezing threshold, but the returned error says that the freezing threshold is much higher now: the freezing threshold 2057103111 cycles.
Updating the number to something larger than 2057103111 should fix the error.

We are working on exposing freezing_threshold_in_cycles via ic.canister_status(), so in the future there will be no need to hardcode the number in transferCycles.

[peterparker]

Indeed @ulan you are right. There is no issue on the network. The problem is that I cannot know the exact freezing threshold in cycles that should be retained and 4_100_ 00 which worked a week ago does not anymore. I set a high value 100_000_000_000 as I had for a long time and I was able to delete the canisters. Thx!

[icme]

One hiccup I ran into while testing this out, that might save others some time.

Because a canister cannot be its own controller, this means that any calls to query the canister_status endpoint must come from that canister’s controller, and then passed to the canister that will be transferring cycles with deposit_cycles.

I did however try in my local environment to incorporate the freezing_threshold field from the ic management canister’s canister_status endpoint and my local environment ended up hanging (stopped). Even trying to stop it returns the following error, which makes me think that either there’s a huge bug in my code or some funky type mismatch occurred with the management canister

Error: The request timed out.

This is a new error for me - I haven’t seen this time out error while developing yet

In fact, I haven’t seen the case where the canister is unstoppable (hangs on Status: Stopping)

byronbecker@Byrons-MacBook-Pro simpleMultiCanister % dfx canister status index
Canister status call result for index.
Status: Stopping
...

Have you or @peterparker been able to get a working example retrieving the management canister’s freezing_threshold?

Also, for reference, my freezing threshold somehow became 148_202_450_254 (much larger than 100_000_000_000)

"IC0501: Canister <canister_id> is out of cycles: requested 311288157983 cycles but the available balance is 411288157983 cycles and the freezing threshold 148202450254 cycles"

[paulyoung]

@icme

Also, for reference, my freezing threshold somehow became 148_202_450_254 (much larger than 100_000_000_000)

Maybe because of this?

Queue + failing heartbeat + stopping canister = death spiral

Oh, sorry @LightningLad91 It’s counter-intuitive, but the freezing threshold is in seconds, not in cycles: The Internet Computer Interface Specification | Internet Computer Home Now the limit can’t be lowered the Canister is controlled by a user, and IC tries to charge the Canister for an ingress message. We’re preparing a patch to unblock you…

[peterparker]

@icme

Have you or @peterparker been able to get a working example retrieving the management canister’s freezing_threshold?

I actually never tried. I followed discussion on “Canister status - Add Freezing threshold in cycles” on the IC spec and waited for the implementation of freezing_threshold_in_cycles which should gives the information we need I think. Did not had time to try it out yet.

[icme]

@paulyoung

Maybe because of this?

Got it, makes sense why it would be that high atm - thanks for the link.

@peterparker

Yep, I gave grabbing the freezing_threshold a brief go, but pivoted to a hacky try/catch solution of bumping up the cycles to leave with the canister (for stop/delete) by 100 billion if the first try fails.

Somewhere in the process of trying to grab the freezing_threshold of canister A through canister A’s controller from ic.canister_status and then feeding it to canister A left my local dfx hung in such a bad state I had to uninstall and kill it, then reinstall. (At least what the code I wrote was testing out, probably a bug on my end).

Was enough of an error to scare me away for the time being, until I have more free time

[simpson]

Does anyone know what the cause of this error is?

Beginning withdrawal of cycles to canister [xxxx]; on failure try --no-wallet --no-withdrawal.
Setting the controller to identity princpal.
Installing temporary wallet in canister Main to enable transfer of cycles.
Transfering 1487618274934 cycles to canister [xxxx].
Error: Failed to delete canister 'Main'.
Caused by: Failed to delete canister 'Main'.
  Failed to deposit 1487618274934 cycles into [xxxx].
    Failed to call update function 'deposit_cycles' regarding canister '[xxxx]'.
      Update call using wallet failed.
        The invocation to the wallet call forward method failed with the error: An error happened during the call: 2: Couldn't send message ````

[Severin]

I’ve never seen such an error.

@simpson

2: Couldn't send message ````

Is that the actual error message or something edited?

[christian]

I’m having exact the same issue with one of my canisters.

@Severin this is just formatting. Here’s how it looks like for me (I removed the cansiter id for privacy reasons):

$ dfx --identity christian canister --network ic delete <XXX>
Beginning withdrawal of cycles to canister <YYY>; on failure try --no-wallet --no-withdrawal.
Setting the controller to identity princpal.
Installing temporary wallet in canister <XXX> to enable transfer of cycles.
Transfering <ZZZ> cycles to canister <XXX>.
Error: Failed to delete canister '<XXX>'.
Caused by: Failed to delete canister '<XXX>'.
  Failed to deposit <ZZZ> cycles into <YYY>.
    Failed to call update function 'deposit_cycles' regarding canister '<XXX>'.
      Update call using wallet failed.
        The invocation to the wallet call forward method failed with the error: An error happened during the call: 2: Couldn't send message

An upgrade of the wallet canister didn’t help.

[Severin]

This issue will be fixed in fix: give more cycles margin when deleting canisters by sesi200 · Pull Request #2784 · dfinity/sdk · GitHub

[timo]

Is this fix in a release yet?

When it says “the wallet does not have enough cycles left in the wallet when deleting a canister” then does “the wallet” refer to the wallet that gets temporarily installed in the canister that is to be deleted?

I compiled dfx from the current main branch and still get the same error.

[Severin]

@timo

Is this fix in a release yet?

It’ll be in 0.12.2-beta.0 that will release today or tomorrow.

@timo

When it says “the wallet does not have enough cycles left in the wallet when deleting a canister” then does “the wallet” refer to the wallet that gets temporarily installed in the canister that is to be deleted?

Yes, that is the temporary wallet.

@timo

I compiled dfx from the current main branch and still get the same error.

That’s a problem then… Are you by chance in a high-replication subnet? The threshold is too low for those network economics. I’ll have to come up with a solution for that