How can top-up be this broken and unstable ?!

I’m getting pretty frustrated with topping up already deployed canisters on the ICP blockchain tbh…
ICP decides to just “eat” the ICP transferred over by using a command like:

dfx ledger --network ic top-up aiekw-byaaa-aaaal-abhwa-cai --amount 2
Transfer sent at block height 4736200
Canister was topped up with 7935200000000 cycles!

While actually showing a success message… My canister cycles currently are only 4.3 trillion (and this top-up alone should have made it 11-12 trillion)…

I even did a “fix” notification by running:

novicavukobratovic@192 ext_v2-master % dfx ledger --network ic notify top-up 4736200 aiekw-byaaa-aaaal-abhwa-cai
Canister aiekw-byaaa-aaaal-abhwa-cai topped up with 7935200000000 cycles

Which ALSO showed success message and did absolutely nothing (did not add 1 cycle to the canister).

What is this, how can it be possible that cycles top-up is this unstable?

1 Like

Thanks for the report. I’ll reach out to the owners of the top-up code.

I see that your canister has an availableCycles() query. It currently returns 6.1T

It seems that some of the cycles were deposited after you saw 4.3T. But that still doesn’t explain why the balance is not 11-12T now. Could it be that the canister transfers cycles to another canister while handling some update call or in a heartbeat?

1 Like

I’ve had a similar experience with creating canisters.

1 Like

Thanks @paulyoung. That indeed looks related.

There is some difference between two issues:

  • Here the top-up command reported success, but the canister’s cycles balance did not increase.

  • In the create-canister thread the command reports an error.

No, nothing was added. I ran the top-up command again with 0.5 ICP and it worked that time, increasing cycles by about 2 trillion cycles. That’s why it has around 6 trillion now. So the 2 ICP did not get converted into cycles at all.

2 Likes

Thanks! That is helpful for the investigation. I’ve reached out to the teams. Most likely they will be able to look into this after the weekend.

To double check: is it possible that during that time the canister was executing messages that send cycles to other canisters? Or drain cycles in any other way?

If the canister is open source and you can share the link, that would help as well.

1 Like

There is zero possibility canister was doing anything at that time. It was EXT v2 collection I’ve deployed for testing on live blockchain and have just deployed it before running the top-up command (to add in more cycles). I can share the code if required to you or the team (the code is not public), but like I said, nothing too specific in the canister code itself, nor was the canister doing anything at the time of using top-up command. This is not the first time this is happening to me with using top-up command. It has not worked before as well (however it was erroring out some months ago) and now it returns a success, deducts ICP tokens and does not deposit cycles (even though it says it did).

2 Likes

Got it. Thanks for confirming!

I see three canisters on that subnet that were created around the time of the report and have aiekw-byaaa-aaaal-abhwa-cai as the only controller:

The sum of the balances in ~7.2T

@AnonymousCoder : could it be that aiekw created these canister programmatically with the initial 2.4T balance around the time of the deposit? Or did you create these canisters manually and make aiekw their only controller?

3 Likes

The collection canister in question (aiekw) does deploy it’s asset canisters dynamically through code. However, the initial “minting” script in node failed due to not having enough cycles to create required asset canisters from the main “aiekw” canister and the script failed, after which I ran the command of top-up to add more cycles to the aiekw canister in order for it to be able to deploy the asset canisters. If those were created at the almost exact time of the issue, I guess it could’ve been possible that these 7.9 trillion cycles went into deploying those 3 asset canisters, but only if it was like at almost exact time, because I did later deposit more cycles and re-ran the minting script again which completed successfuly

1 Like

Does the sum of all cycles you deposited to aiekw match the sum of balances of all four canisters?

1 Like

Can’t tell anymore since they’ve been out for like 6-8 hours now, probably less than initial deposited amount if any :slight_smile: But if I put together your info about balances of the 3 asset canisters it does seem to add up very close to 7.9 trillion that it said were deposited, although it is around 7.2 trillion when added together so I am not sure.

1 Like

Cool! Thanks for checking. We are missing 0.7T = 7.9T - 7.2T cycles. There is a fee of 0.1T for creating a single canister, which explains 0.3T. After taking that into account we are missing 0.4T. I guess some of it was burned by execution and periodic charges for memory usage. Shall we consider the remaining as measurement noise? :slight_smile:

7 Likes

Yeah I think that in this case it def might be possible that the canister took the deposited cycles and immediately made the asset canisters as soon as cycles were deposited and that’s the reason i never seen that 7.2 trillion cycles increase after the deposit. I will definitely keep my eyes open on this command in the future and report any issues if I find them! Thank you for shedding some light on this issue, much appreciated!

2 Likes

You’re welcome! Please feel free to tag me if you see and report similar issues.

4 Likes

@ulan is a superhero

10 Likes