Title: Designing around subnet replica upgrade windows — advance notice, stateful failover, and timers
Setup. A small booking product on the IC: public website canisters serving HTML and a React bundle, and separate stateful backend canisters holding catalogue, availability and orders, talking to a payment gateway over HTTPS outcalls. Four canisters across three subnets — io67a-…-gqe and ejbmu-…-zqe (verified_application) and e66qm-…-5qe (application).
What we saw. A deploy to io67a-…-gqe failed with no_healthy_nodes, and asset requests to that canister returned {“error_type”: “subnet_updating”}. Measured from our side:
23:08:32 asset fetch OK (1,278,114 bytes)
23:16:29 503 subnet_updating
23:19:23 200, stable
So under three minutes of observed unavailability — consistent with the “around two minutes” figure in DFINITY’s own material, so nothing looks wrong here. What caught us out was everything around it.
From a visitor’s point of view the site looked up: the HTML document came from the boundary node cache and returned 200, while anything that had to reach a replica failed. A customer in that window sees a working site whose booking button does nothing.
I have read the docs on deploying to a specific subnet, so I know placement is selectable at creation with --subnet / --proxy. My questions are the ones I could not find answers for.
-
Can an operator see it coming? Replica versions are elected and rolled out by NNS proposal, but is there anything that says this subnet, this window — a feed, an API, a status page — that we could subscribe to? Today we would learn about it from a failing deploy or a customer complaint. Is watching proposals genuinely the only signal, and how frequently does a given subnet get upgraded in practice?
-
What do people do for a stateful backend? We can place our two backends on different subnets so one window cannot take both down — but that only works because they serve independent destinations. It is not redundancy for a single destination. Running one destination’s backend on two subnets means splitting or replicating state, each with its own failure modes, to cover a few minutes a handful of times a year. Is anyone doing that in production, or is the accepted answer to make the client retry and ride it out?
-
And the frontend? A site canister is nearly stateless, so mirroring it on a second subnet and failing over client-side seems straightforward — except the client has to load something first in order to know to fail over, and that first thing is exactly what may be unavailable. Is there an established pattern, or does everyone rely on the boundary node cache?
-
Do canister timers survive a subnet replica upgrade? The docs are explicit that timers are cleared on canister upgrades and must be re-armed in post_upgrade. I could not find whether a subnet replica upgrade preserves them. We use a timer to chase unconfirmed payments, so a silently dropped one means a customer charged with no booking recorded.
Happy to share more detail on any of it.