How Canic could solve this
I agree that the replica should ultimately preserve an overdue timer and reactivate it
when the canister becomes runnable again.
The IC exposes one global timer per canister. Timer libraries maintain their task
queues in Wasm memory and use that global timer as the wake-up mechanism. The
specification currently deactivates the global timer when a canister runs out of
cycles and does not reactivate it after a top-up. The IC documentation describes this
behavior explicitly.
( https://docs.internetcomputer.org/guides/security/canister-upgrades/#reinstantiate-timers-during-upgrades )
Canic could provide a practical recovery mechanism without rebuilding or upgrading the
canister.
1. Add a canic_timer_wake endpoint
Every Canic canister could expose a small, idempotent recovery endpoint which:
- authenticates the Canic root, controller, DAO, or configured guardian;
- calls ic0.global_timer_set(now) to reactivate the global timer;
- lets the existing Rust CDK timer queue continue processing;
- returns a report containing the previous deadline, wake time, and timer status.
In the normal frozen-but-not-uninstalled case, the Wasm heap and CDK timer queue still
exist; only the protocol-level global timer has been deactivated. Therefore, rearming
the global timer should recover the existing timers without changing the Wasm hash.
The recovery path would become:
top up canister
→ call canic_timer_wake
→ global timer fires
→ existing CDK timer queue resumes
No source checkout, dependency upgrade, rebuild, or governance proposal to install new
Wasm would be required.
2. Couple every Canic top-up with a wake call
Canic’s funding workflow could treat these as one recoverable operation:
- Deposit cycles.
- Confirm that the canister has enough liquid cycles.
- Call canic_timer_wake.
- Verify that the timer scheduler is running again.
The workflow should be retry-safe, so if the funding canister crashes between steps 1
and 3, it can resume from the persisted receipt without depositing cycles twice.
For manual recovery, Canic could also offer something like:
canic medic recover-timers
That command could wake every Canic canister in the deployment and report which timers
resumed. This alone would replace the current “modify something, rebuild everything,
and upgrade every canister” procedure.
3. Add a fleet-level cycles guardian
Prevention should not depend only on a timer inside the canister that is about to
freeze.
A Canic root or dedicated guardian canister could monitor managed children, top them
up before their freezing reserve is reached, and always send the wake call after a
recovery top-up. The root itself would still need an independent guardian—another
canister or an operator service—because a canister cannot rescue itself once it no
longer executes.
For a DAO deployment, the guardian could be explicitly authorized by configuration, or
governance could call the recovery endpoint as a generic function. That is much
lighter and safer than approving a Wasm upgrade merely to restart timers.
4. Make critical timers durable
For stronger guarantees, Canic should eventually treat CDK timers only as wake-up
signals, not as the source of truth.
Each critical timer would have a stable logical record containing:
- a stable timer name;
- its interval and next due time;
- whether it is enabled;
- its last successful execution;
- an in-flight lease or generation;
- its missed-tick policy.
On init, upgrade, or canic_timer_wake, Canic would reconcile those stable records with
the in-memory scheduler. After a long outage, each timer could choose whether to skip
missed ticks, run once immediately, or perform bounded catch-up. The default should
probably be “run once, then resume the normal cadence” to avoid five months of missed
executions firing at once.
Important limitation
Canic cannot make an arbitrary top-up execute canister code by itself. Adding cycles
is a management operation and does not invoke a canister lifecycle hook. Without a
protocol change, some live actor must send the first post-top-up wake call.
So I see two complementary fixes:
-
Platform fix: preserve the global timer while frozen and automatically requeue it
when cycles make the canister runnable again.
-
Canic fix: provide durable timer intent, an idempotent wake endpoint, and funding/
guardian workflows that automatically call it.
The platform fix is still the ideal answer, but the Canic approach could remove almost
all of the operational pain immediately—and, most importantly, eliminate the need to
rebuild and upgrade an old canister just to restart its timers.
