# Queue + failing heartbeat + stopping canister = death spiral
**URL:** https://forum.dfinity.org/t/queue-failing-heartbeat-stopping-canister-death-spiral/13328
**Category:** Developers
**Created:** [May 28, 2022, 4:58am UTC](https://forum.dfinity.org/t/queue-failing-heartbeat-stopping-canister-death-spiral/13328 "2022-05-28T04:58:34Z")
**Posts on this page:** 20
**Page:** 1
### Author:  [@bob11](https://forum.dfinity.org/u/bob11)
#### Post date: [May 28, 2022, 4:58am UTC](https://forum.dfinity.org/t/queue-failing-heartbeat-stopping-canister-death-spiral/13328/1 "2022-05-28T04:58:34Z")
Here are steps to replicate:
1. Set up heartbeat to run a specific function (pulling from a queue) every time it runs
2. Make sure that the process fails
3. Turn on heartbeat. Heartbeat will now run forever trying to run the job in the queue, but will forever fail.
4. Try to stop the canister. Canister won’t stop because heartbeat has callbacks. But all requests from this point forward will fail and say “canister stopping”
5. At this point, you cannot call the canister at all (because canister is stopping), the canister will never stop (it can’t because heartbeat is running continuously), and heartbeat can’t stop because there is an item in the queue that keeps failing.
This is what I am affectionately calling the canister death spiral.
I know this may not be the best way to implement a queue with heartbeat, but it seems like there should be something I can do to rescue a canister in this state. Current best idea is wait for the canister to run really low on cycles (at the freezing threshold) and then regain control of the canister.
Any other thoughts/ideas?
---
### Author:  [@paulyoung](https://forum.dfinity.org/u/paulyoung)
#### Post date: [May 28, 2022, 5:05am UTC](https://forum.dfinity.org/t/queue-failing-heartbeat-stopping-canister-death-spiral/13328/2 "2022-05-28T05:05:39Z")
Upgrade the canister to remove the heartbeat function?
---
### Author:  [@bob11](https://forum.dfinity.org/u/bob11)
#### Post date: [May 28, 2022, 5:10am UTC](https://forum.dfinity.org/t/queue-failing-heartbeat-stopping-canister-death-spiral/13328/3 "2022-05-28T05:10:32Z")
Tried it. Can’t upgrade the canister. Got this error:
`error: code 5, message: "Canister 4fcza-biaaa-aaaah-abi4q-cai trapped explicitly: canister_pre_upgrade attempted with outstanding message callbacks (try stopping the canister before upgrade)"`
---
### Author:  [@paulyoung](https://forum.dfinity.org/u/paulyoung)
#### Post date: [May 28, 2022, 5:11am UTC](https://forum.dfinity.org/t/queue-failing-heartbeat-stopping-canister-death-spiral/13328/4 "2022-05-28T05:11:34Z")
Under “Canister Status”, the [spec](https://internetcomputer.org/docs/current/references/ic-interface-spec/) says this:
> In all cases, calls to the [management canister](https://internetcomputer.org/docs/current/references/ic-interface-spec/#the-ic-management-canister) are processed, regardless of the state of the managed canister.
The management canister is the one that can install
code and perform upgrades.
---
### Author:  [@paulyoung](https://forum.dfinity.org/u/paulyoung)
#### Post date: [May 28, 2022, 5:12am UTC](https://forum.dfinity.org/t/queue-failing-heartbeat-stopping-canister-death-spiral/13328/5 "2022-05-28T05:12:56Z")
I guess using one way functions would have prevented this. That doesn’t help now though.
---
### Author:  [@paulyoung](https://forum.dfinity.org/u/paulyoung)
#### Post date: [May 28, 2022, 5:14am UTC](https://forum.dfinity.org/t/queue-failing-heartbeat-stopping-canister-death-spiral/13328/6 "2022-05-28T05:14:53Z")
Depending on the data involved, maybe you can `reinstall` instead of `upgrade`. You would lose all state though.
---
### Author:  [@paulyoung](https://forum.dfinity.org/u/paulyoung)
#### Post date: [May 28, 2022, 5:16am UTC](https://forum.dfinity.org/t/queue-failing-heartbeat-stopping-canister-death-spiral/13328/7 "2022-05-28T05:16:57Z")
Not sure there’s much benefit to _reinstalling_ over _uninstalling_ other than I don’t remember if `dfx` has a command to uninstall.
---
### Author:  [@bob11](https://forum.dfinity.org/u/bob11)
#### Post date: [May 28, 2022, 5:17am UTC](https://forum.dfinity.org/t/queue-failing-heartbeat-stopping-canister-death-spiral/13328/8 "2022-05-28T05:17:35Z")
Yeah, we really need to preserve state, so hoping to not have to uninstall/reinstall
---
### Author:  [@paulyoung](https://forum.dfinity.org/u/paulyoung)
#### Post date: [May 28, 2022, 5:20am UTC](https://forum.dfinity.org/t/queue-failing-heartbeat-stopping-canister-death-spiral/13328/9 "2022-05-28T05:20:11Z")
What’s the nature of the failure with processing the queue? Does the queue live in another canister?
I’m wondering if you can upgrade that instead to address the problem from that side.
---
### Author:  [@bob11](https://forum.dfinity.org/u/bob11)
#### Post date: [May 28, 2022, 5:27am UTC](https://forum.dfinity.org/t/queue-failing-heartbeat-stopping-canister-death-spiral/13328/10 "2022-05-28T05:27:23Z")
Queue lives on the same canister, so can’t upgrade on that side either
---
### Author:  [@paulyoung](https://forum.dfinity.org/u/paulyoung)
#### Post date: [May 28, 2022, 5:35am UTC](https://forum.dfinity.org/t/queue-failing-heartbeat-stopping-canister-death-spiral/13328/11 "2022-05-28T05:35:46Z")
Does `canister_pre_upgrade` here only mean there’s a `pre_upgrade` function defined in your code?
If so, what happens if you remove both that and the heartbeat function and then try to upgrade using that?
---
### Author:  [@paulyoung](https://forum.dfinity.org/u/paulyoung)
#### Post date: [May 28, 2022, 5:43am UTC](https://forum.dfinity.org/t/queue-failing-heartbeat-stopping-canister-death-spiral/13328/12 "2022-05-28T05:43:46Z")
If that doesn’t work, you should be able to downgrade the Motoko compiler to a version before this pull request and recompile your canister using that.
>
>
> Fixes #2676
>
> Although this appears to do the right thing for the replica (dru…n), it seems superfluous for ic-ref.
>
> @nomeata does ic-ref already wait for the call back queue to drain before doing an upgrade?
When you upgrade again you won’t get the `canister_pre_upgrade` error.
---
### Author:  [@paulyoung](https://forum.dfinity.org/u/paulyoung)
#### Post date: [May 28, 2022, 7:49am UTC](https://forum.dfinity.org/t/queue-failing-heartbeat-stopping-canister-death-spiral/13328/13 "2022-05-28T07:49:01Z")
If downgrading is problematic you should be able to remove the relevant lines of code and build the Motoko compiler yourself.
I could do that and send you a build if you can tell me which version you’re using.
I’d also need to know some info on your system architecture. I’m on an M1 MacBook Pro so a similar device would be the most convenient. I also have a Linux VM that I can probably use if necessary.
---
### Author:  [@diegop](https://forum.dfinity.org/u/diegop)
#### Post date: [May 28, 2022, 4:09pm UTC](https://forum.dfinity.org/t/queue-failing-heartbeat-stopping-canister-death-spiral/13328/14 "2022-05-28T16:09:17Z")
Let me ping folks internally to see if anyone has any other ideas
---
### Author:  [@levi](https://forum.dfinity.org/u/levi)
#### Post date: [May 28, 2022, 8:26pm UTC](https://forum.dfinity.org/t/queue-failing-heartbeat-stopping-canister-death-spiral/13328/15 "2022-05-28T20:26:09Z")
Seems to me it is better if the heartbeat stops waking up as soon as the canister is put into the stopping mode.
---
### Author:  [@torates](https://forum.dfinity.org/u/torates)
#### Post date: [May 29, 2022, 12:33am UTC](https://forum.dfinity.org/t/queue-failing-heartbeat-stopping-canister-death-spiral/13328/16 "2022-05-29T00:33:28Z")
Hey Paul, I’m working with Bob through this issue. The canister is controlled by my `default` identity on dfx.
Also, I have a M1, but mine is a MacBook Air 2022, running `dfx 0.9.3`. Any clue on how to re-build the motoko compiler and use it instead of my current compiler?
Your help is much appreciated.
---
### Author:  [@paulyoung](https://forum.dfinity.org/u/paulyoung)
#### Post date: [May 29, 2022, 12:41am UTC](https://forum.dfinity.org/t/queue-failing-heartbeat-stopping-canister-death-spiral/13328/17 "2022-05-29T00:41:18Z")
It required some non-trivial tweaks but I did a custom build of Motoko recently using a variation of the branch linked here: [nix-shell failure · Issue #3041 · dfinity/motoko · GitHub](https://github.com/dfinity/motoko/issues/3041#issuecomment-1125688621)
I should easily be able to do a build of 0.9.3 that removes the trap in the pre upgrade hook based on that. You’d be trusting that I’m not doing anything malicious in a build I send you though.
If you want to try it yourself and you’re familiar with nix, using [GitHub - ninegua/ic-nix: Build Internet Computer projects with Nix](https://github.com/ninegua/ic-nix) might be easier. Then it would be a case of following the instructions here: [motoko/Building.md at master · dfinity/motoko · GitHub](https://github.com/dfinity/motoko/blob/master/Building.md#development-using-nix)
---
### Author:  [@levi](https://forum.dfinity.org/u/levi)
#### Post date: [May 29, 2022, 1:13am UTC](https://forum.dfinity.org/t/queue-failing-heartbeat-stopping-canister-death-spiral/13328/18 "2022-05-29T01:13:24Z")
The pre-upgrade hook gets run on the module that is already running in the canister. only the post-upgrade hook is called on the new module. There is no way to upgrade the canister if the pre-upgrade hook fails. The motoko code that traps is in the pre-upgrade hook. the only way to upgrade that canister (without reinstalling or uninstalling) is to stop all the pending callbacks. If the heartbeat keeps waking up when the canister is stopping that seems like a bug to me.
---
### Author:  [@paulyoung](https://forum.dfinity.org/u/paulyoung)
#### Post date: [May 29, 2022, 1:29am UTC](https://forum.dfinity.org/t/queue-failing-heartbeat-stopping-canister-death-spiral/13328/19 "2022-05-29T01:29:01Z")
That would make sense.
In that case the only way to address this might be to propose a change to the way heartbeat works and wait for that to land.
---
### Author:  [@torates](https://forum.dfinity.org/u/torates)
#### Post date: [May 29, 2022, 1:14pm UTC](https://forum.dfinity.org/t/queue-failing-heartbeat-stopping-canister-death-spiral/13328/20 "2022-05-29T13:14:43Z")
In any case, I think the following error message should be reformatted to avoid this issue in the future:
```auto
Error: The Replica returned an error: code 5, message: "Canister trapped explicitly: canister_pre_upgrade attempted with outstanding message callbacks (try stopping the canister before upgrade)"
```
Maybe include a clause to not try the stop the canister if the user is using heartbeat in the current state?
[Next page](https://forum.dfinity.org/t/queue-failing-heartbeat-stopping-canister-death-spiral/13328.md?page=2)