What does it mean when nearly all nodes in the European Subnet are marked as degraded?
That they waiting for the node rewards?
I wasn’t aware that these nodes are marked as degraded but I think we know what’s going on.
We accidentally introduced a regression around burn_cycles. Previously, if you tried to burn more cycles than you had, it would burn the amount you had. With the regression, the call would instead fail. We quietly fixed that bug last week, but it will only roll out to all subnets next week.
It appears that the only subnet where this actually happened is the European subnet, maybe some old BOB canister running of of cycles.
Nodes are marked as degraded, because our internal alerting actually caught that bug, and because it’s a deterministic bug, the alert fired on all nodes simultaneously. But beyond the alert and the slightly different behaviour of the burn_cycles endpoint there is no real damage, and it will be fixed by the rollout of the fix.
I think it could be due to the responsibility of replacing nodes now falling to the NPs.
Potentially relevant discussions