Hotfix →
Looks like a divide by zero error was introduced due to subnet_schedule.len() differing to state.canister_states().len(), and the division here. I guess this recurs after a restart so the subnet stalls.
I’m not clear why this would cause degraded performance on other subnets (rather than just fully fledged stalling), so it may actually be unrelated to what’s being experienced on this particular subnet (lhg73). I’ll avoid speculating further
I think this change was mainly done to make the code more robust and cover some rare edge cases like newly created subnets with no canisters on them, that division has been there for many months and the number_of_canisters value hasn’t been changed, so it’s unlikely to have caused the downtimes.
subnet_schedule and canister_states lengths diverging, with the former being greater, could have played a role. Generally they are kept in sync throughout the codebase, but here they could have different values cause canister_priority_mut is called for all canister ids that were executed in the round and it uses subnet_schedule::get_mut, which adds a default entry in the priorities map if one doesn’t exist for the provided id. So if a canister was deleted in the round but still registered in the executed set it would be added back to the subnet_schedule.
so, what’s up with lhg73? still broken, been hours
I can’t trade either, what’s going on
Could any member of the DFINITY team please help notify the relevant devs? The lhg73 subnet appears to be down — it has been showing 0 Cycles/s for quite some time.
Bump @DFINITYTeamMembers
Does anyone care to explain why this 16+ hours downtime is NOT mentioned on the https://status.internetcomputer.org ?
lhg73-sax6z-2zank-6oer2-575lz-zgbxx-ptudx-5korm-fy7we-kh4hl-pqe
Authorized only, shouldnt it be Public subnet?
Historically it became private because the memory usage on that subnet surpassed certain thresholds.
Those thresholds are now significantly higher, so based on that metric this subnet could be made public today.
cc @alexu
Not sure what the current issue is that apparently only affects some of the 25K canisters, there were a few deployed on 27th and there was some testing or smth as shown by the roughly 14 GB temporary state increase but even that dropped back.
Nodes seem to be healthy, only thing is to wait for the hotfixed replica to be deployed I guess ?
zzz, any update? been 24h since subnet is down
Update:
Several devs have responded to us on Slack. With their investigation and support, our dev team expects to fully restore normal trading soon.
We are in contact with the ICPSwap team helping them debugging their canister.
The subnet is not down. Let me give some more context on the subnet metrics you are observing. The increased state size during the stall of the lspz2 subnet yesterday, the increased cycle burn during the same time period and the (supposed) 0 cycle burn since then are all the the same effect. When a canister does a XNet call, both memory and cycles are reserved while waiting for the response. Yesterday, any calls to lspz2 were stuck. During that time, both the cycles and the memory were therefore withheld, and only refunded when the subnet got unstuck. As soon as lspz2 was back online, the memory was refunded, as you can see in the graphs that @ZackDS shared. For the cycles, an unfortunate quirk of the public dashboard is that it does not show cycle refunds as negative cycle burn. Instead, it shows 0 cycle burn until the refunded amount is properly burned. It’s more obvious what is happening from the internal dashboards:
As you can see here, the cycles consumed sharply drops (due to the refunds) and the subnet has been burning more cycles ever since. Unfortunately the public dashboard will show 0 burn until the total cycles consumed exceeds the historic peak again. But crucially, that does not mean that the subnet is down.
It is however true that ICPSwap did not fully recover after the other subnet came back online, which is likely an effect of the uncommon errors their canisters received during the stall of the lspz2 subnet. We are trying to help the ICPSwap team to understand what exactly happened to their canisters and how to fix it.