What is happening with the cycles minting canister?💆🏻‍♂️

There’s this issue with a new project that seems like requires computational power but many users having issues minting, also
The subnet it run presents a limitation in terms of cycle burn rate @bjoern explained it’s because of hardware specifications and that cycle price for resources it’s fixed. But I’m wondering if this is good or bad, good because now dfinity engineers are seeing how there’s an issue or bottleneck on the cycles minting canister meaning demand for computing power is increasing, but bad because don’t know if this can be solved, I really hope so…

Also I did some numbers with the subnet at the specification is today:

37B cycles per second seems to be the limit (per subnet)

37B / 1T cycles = 0.037 cents per second
0.037 * 60seconds= 2.22 Usd per minute
2.22 USDper minute *60 minutes = 133.2 USD hour
133.2 USD hour * 24hour = 3.196 USD a day

3.196 USD * 30 days= 95.900 USD a month

Node rewards per subnet let’s say 3k a month for each node, there are 13 nodes= 39.000 USD

So 95.900 - 39.0000 = 56.900 profit? This if the subnet it’s on its max capacity, without counting storage of course.

2 Likes

There are 1.214 miners (canisters) waiting to be created and be assigned cycles………

This product was gaining traction now with all
Those issues people will bet upset. :man_facepalming: @bjoern @bjoernek

2 Likes

Hello @bjoern @dominicwilliams

It seems there is a bug in their application canister which stops them from making progress. It is still not entirely clear why the canister creations were delayed earlier, but this was definitely not caused by a limit in the cycles minting canister.

4 Likes

Same issue, lost 6 icp gained nothing

We’re using notify_top_up from cmc, most of the requests get NotiftyError::Processing, which seems to mean that it’s already processing the request.

2 Likes

If you call notify_top_up repeatedly with the same ledger block height, then the first call will initiate the top up, and all subsequent calls will be responded with NotifyError::Processing to indicate that there is already a conversion ongoing. Do you see these things taking very long (ie. in the case of the canisters on e66qm more than a few minutes or so)? Can you provide an example (best would be ledger block height and canister that is supposed to be topped up)?

You can see it happen repeatedly: https://dashboard.internetcomputer.org/account/6b896884e0b42634eca9c68c435c47b0ef2b97cf874a17198856b9c4efe89249

This is the account that corresponds to our canister. Burn happens infrequently and extremely slowly. We get NotifyError::Processing from all the notify_top_up call

I see that there are very few burn transactions but that does not help me in understanding where the bottleneck is. What is the canister id you’re topping up?

yhz26-biaaa-aaaal-qjtsq-cai

2 Likes

Thank you. We can now validate that there is an issue, we are now trying to understand the root cause. Status page has been updated.

2 Likes

It seems that now things to work, if I now trigger notify_top_up manually, things seem to proceed:

% dfx canister --ic call rkp4c-7iaaa-aaaaa-aaaca-cai notify_top_up '(record {block_index = 13832388; canister_id = principal "yhz26-biaaa-aaaal-qjtsq-cai";})'
(variant { Ok = 5_324_400_000_000 : nat })

So I cannot reproduce the problems at the moment.

3 Likes

I would strongly disagree, no one is making calls and the canister is burning ICP as I am writing: https://dashboard.internetcomputer.org/account/6b896884e0b42634eca9c68c435c47b0ef2b97cf874a17198856b9c4efe89249 There is for sure something off.

@bjoern subnet now seems to be stuck at 7B cycles per second…

Yesterday people were saying it was at 37B
Today instead of this being increased, it’s decreasing why?

Today more
ICP were burned on that subnet so why yesterday hit 37B and now can’t go through 7B …?

Babble posted about making the notify calls. But also that there’s still some 858 ICP unaccounted for. This likely needs a deeper analysis of the transactions, could also be that so far only notify_top_up was tried not notify_create_canister.

In short: the ~37B are a glitch on the dashboard. Those are overcounted because canisters have to pay in advance for some operations (e.g. making a call to another canister, the calling canister pays for the call but also for the maximum size of the reply, and gets a refund if the reply is smaller than max). The exact metric used by the dashboard does not account for these refunds. The 37B spikes were in periods where bob canisters were making lots of outbound calls, and so a large fraction of the cycles were later refunded.

The ~7B we see right now are probably pretty close to what a singe 13-node subnet is currently burning under full computational load.

We still are encountering many issues regarding canister management: stop canister, install canister, upgrade canister, reinstall canister etc. All the canisters we deployed on this subnet seem to be in limbo, any call to https://dashboard.internetcomputer.org/canister/23tio-diaaa-aaaal-qjyma-cai will timeout for example.

Sorry If i didn’t understand it correctly, so you are saying the maximum burn rate that subnet can have is 7B cycles per second?

This means 18.144 USD per month. How is the subnet economically sustainable or profitable if each node on average its paid 2500 USD * 13= 32.500 USD for the whole subnet.

But the subnet can just burn 18.144 ?? Am I correct or am not understanding? (Hope so)