Since today, I have noticed that update requests on the European subnet are either not processed or take a very long time before eventually timing out.
Reject text: Ingress message 0x29eb1b70336f98a1a811cff34f4fe9e0f45421216799203118d38547701d6d06 timed out waiting to start executing.
You can test here: https://hengx-riaaa-aaaas-ajw5a-cai.icp0.io/
Just saw this and am not involved so I can’t say what’s happening, but engineers are on it and there will be a rollback proposal soon for the subnet
Here’s the proposal: Proposal: 140725 - ICP Dashboard
Looks like it solved at least this issue. Thanks for the heads up.
It seems to be working again now. What was the issue?
AFAIU it’s nothing obvious so far, and as long as we’re not certain that the issue could not be abused (anymore?) I’ll refrain from posting the ideas I see discussed
Maybe will have a post mortem, like in the good old days but wouldn’t keep my hopes up.
Strange to see only 1 subnet being affected (so far) just like with latest lspz2 subnet.
https://forum.dfinity.org/t/subnet-management-lspz2-application/35322/99?u=zackds
Understandably, I need to explain to my company why this outage occurred, so I would appreciate some additional information.
Please feel free to contact me via DM as well.
For context, we run a business application for our daily operations on the European Subnet, so outages like this are immediately noticeable. One of the main reasons we rely on this infrastructure is its promise of being unstoppable.
Totally understandable. I got the green light for some explanation.
We are currently improving the replica to support more canisters per subnet. One of the ways to do this is to do less work on inactive canisters. Refactorings are happening that will stop inactive canisters from being scheduled or accessed when not needed.
Bug: a code path different from the one refactored (…at least if I understood the explanation correctly) suddenly put a bunch of canisters that are frozen (so they should not be scheduled for execution) into the list of canisters that need to be scheduled. Therefore way too many canisters were available to the scheduler, and the canisters that had actual work to do were lost in round robin somewhere. Apparently the European subnet has particularly many of those, so it was the subnet most affected.
Remember how during the big BOB craze a while back nobody could get messages through? It was the same situation, just with no actual work executed (most scheduled canisters are frozen, therefore no calls are actually run), and therefore the block rate did not drop like in the BOB times.
The following may sound like I’m trying to bullshit you into believing that everything was fine, even though it wasn’t. I’m actually trying to explain the guarantees the system promises a bit so you understand the limitations better and you can plan for more scenarios.
Technically nothing was stopped. Live canisters were still available, and if you tried to run only query calls you probably wouldn’t have noticed anything. All the bug did was ‘just’ (lol) simulate what happens under extreme load. If a huge number of canisters on the same subnet receive messages (or schedule timers, or have heartbeats) then the scheduler IIRC goes round-robin through all canisters and lets them run a message whenever it gets to the canister. If there are too many to go through before your ingress message times out, then many calls will fail.
One thing you can do in such resource contention situations is to use the canister setting compute_allocation (best explanation I found in the spec is in this section). It is quite expensive, but 1% compute allocation guarantees that your canister will be scheduled every 100 rounds, so every 50ish seconds as long as the block rate is around 2/s. Incidentally, with this particular bug compute allocation would also have helped.
Another idea I just had (so I haven’t thought much about how useful it would be) is that canister-to-canister messages that are not bounded-wait will not time out. So if you send the message to a canister on a different subnet first, it will not be affected by the resource contention, and can send it to the subnet under load with no expiry time. (May need retry logic if the subnet does not get around to accepting the message in time the first time around.) You may have to wait for a looong time, but the message will not time out and will be processed at some point. Of course if you proxy through a different canister first you lose e.g. the caller principal, but maybe that doesn’t matter in certain scenarios.
Given that all other subnets are being reverted to de43a37 a HOTFIX and a post mortem is expected.
Again thank you for sharing.
Thank you very much, and thanks also to the team for fixing the issue so fast.
I also think a post mortem would be appropriate if there’s anything to learn, but I don’t think a hotfix is needed. Reverting to the previous version removes any chance of exploitation, and the patch should be merged any moment now, so next week’s release will be safe from this issue
Update: Peolpe involved just promised that there will be a post mortem
bkfrj)On March 5th, 2026, the European Subnet (bkfrj) experienced excessive latency (on the order of 30 minutes) after being upgraded to replica version 781ef50. The issue was caused by a scheduler refactoring touching behavior not covered by tests (frozen canisters being charged scheduler priority for messages they cannot execute). Things returned to normal after the subnet was reverted (to replica version de43a37).
One affected subnet (bkfrj).
The incident lasted for 2.5 hours and was solved by simply reverting the replica upgrade.
Apart from automated traffic generated every 4 minutes or so by timers, the subnet gets very little traffic (on the order of 0.5 transactions per second).
Fewer than 0.1 ingress messages per second expired (just under 20% of the ingress traffic).
Key Events (all times UTC; delay since incident start in parenthesis)
2026-03-05:
10:58: Incident start (subnet bkfrj upgraded).
11:48: Issue Detected. (+0:50)
11:48: Issue reported by users on forum. (+0:50)
12:22: Inspection of Execution Metrics dashboard shows that timers are slow to fire (+1:24)
12:46: Suggestion to roll back the upgrade (+1:48)
12:47: Other subnet upgrades paused (+1:49)
12:59: Decision reached to roll back the upgrade (+2:01)
13:13: Rollback proposal out (+2:15)
13:23: Rollback complete, behavior returns to normal (+2:25)
14:35: Incident closed (+3:36)
Once the decision to roll back the replica upgrade was made, it took less than 25 minutes to get the subnet back into fully functioning state. This obviously only works if the subnet is not stalled, but it’s something to keep in mind.
The issue was introduced while refactoring the canister scheduler to be more efficient when scheduling relatively few active canisters from among a huge number of hosted canisters (by far the most common case). The existing scheduler (actually better described as a mix of control loop plus scheduler) is monolithic.
And the tests, while sizable (6.6k LOC) all test what is essentially end-to-end behavior. They are also an organic collection that grew over time, so there is very little structure to them. It is hard to tell what is being tested and what not.
The breaking change touched an aspect of the control loop logic (canisters without enough cycles to actually execute anything) that, while hinted at in a couple of comments, was entirely untested. But the sheer volume of tests and their lack of structure made it impossible to make this determination.
In the same vein, tests that cover old, organically grown code with new, differently behaved code are difficult to write. For one, it can be (and in this case it is) very hard to fully grasp all the explicit and implicit behaviors of the old code. However, writing some high-level tests ahead of time and ensuring at least nominal coverage of all existing code would have likely helped.
Subnet bkfrj has over 4000 (potentially quite a bit more) canisters with timers and without enough cycles to execute anything. Because these timers all triggered a long time ago and were never executed, they are enqueued for execution every single round. Meaning that every round (with or without a broken scheduler) ends up “executing” 4000 timers (by doing absolutely nothing with them) and terminating.
Ensure comprehensive scheduler testing before proceeding further with the refactoring (see #9270, #9307, #9329, #9343, #9351, #9381, #9402, #9438, #9457).
Subnet bkfrj currently has over 4k canisters with heartbeats but not enough cycles to execute them (or anything else). This means that even in the absence of any other load, the subnet reaches the 8B instruction limit and stops executing anything else simply from the 2M instruction overhead that we account for every (executed or not) message.
Aiming for efficiency while charging for “full round executions”, the breaking change switched from iterating over all canisters and checking for “nothing left to execute” to only iterating over canisters that had actually completed at least one message execution during the round. The 4k canisters above however, all had something to execute at the beginning of each round; nothing left to execute at the end; and had not executed anything in-between. Comments in the code mention “canisters with too few cycles” being unable to actually execute messages (and there’s even a return value for execute_canister() signifying “nothing was executed because there weren’t enough cycles”); but only one test touched on this behavior and only to the extent that the message was not executed.