Infinite Loop preventing backend canister from stopping need help ASAP

Developers
1

My app is in a state where users don’t have access to their canisters for the purpose of upgrading their canister code. To perform this upgrade, it requires that I upgrade the canister code in the backend canister and now of all times, the backend canister refuses to stop-- suspending the app in a state of limbo. I need help resolving this sooner rather than later so that users can carry on. @diegop could you please refer me to the person who would best be able to help me resolve this issue.

the error thrown in the console is as follows:

Error: Failed to stop canister hxx6x-baaaa-aaaap-qaaxq-cai.
Caused by: Failed to stop canister hxx6x-baaaa-aaaap-qaaxq-cai.
  Failed to call update function 'stop_canister' regarding canister 'hxx6x-baaaa-aaaap-qaaxq-cai'.
    Update call (without wallet) failed.
      The request timed out.

edit: I’ve identified the cause of the issue. One of the methods called within the heartbeat method has a while loop that loops over an index, but I forgot to increment the index- causing an infinite loop to be called within the heartbeat method. How do I terminate this method?

1 Like
2

Can you try raising the canister freezing threshold like was done in this thread?

1 Like
3

@icme I was able to successfully raise the freezing threshold, but now I’m unable to make any changes to the canister status nor the code because the canister is now frozen. when i attempt to do either, I get the following error:

Error: Failed to stop canister ...
Caused by: Failed to stop canister ...
  Failed to call update function 'stop_canister' regarding canister '..'
    Update call (without wallet) failed.
      The replica returned an HTTP Error: Http Error: status 503 Service Unavailable, content type "application/cbor", content: Canister ... is out of cycles: requested 1312000 cycles but the available balance is 375780597646572 cycles and the freezing threshold 177456601851851851 cycles
4

Can you now try to lower the threshold to its previous value and then immediately stop the canister (i.e concat the two commands together)?

Hopefully the loop was killed so you can stop your canister before any of your user’s requests re-trigger it.

1 Like
5

Let me ping some folks @Jesse

6

@icme, I did that and it worked!. i was able to stop the canister. Crisis averted. @diegop , no need. thank you both for responding quickly.

5 Likes
7

Awesome! :partying_face:

@diegop

This time, we were able to solve the issue, but I’d imagine that for a high traffic application or one where the infinite loop is baked more deeply into the canister it might be more difficult.

It would be nice to have a single command type of mechanism similar to raising the freezing threshold, lowering it to its original value, and then stopping the canister that would guarantee the first action taken after lowering the freezing threshold would be for the canister to be stopped (nothing could re-trigger the loop).

3 Likes
8

thank you, let me escalate this up.

1 Like
9

I think this mechanism isn’t strictly necessary actually. It’s possible to stop a frozen canister, but the trick is that another canister needs to send the stop message so that the frozen canister doesn’t need to pay for it. So the following steps would always work and don’t require any atomic combining of requests:

  1. Update the canister setting to raise the freezing threshold and add some other canister as a new controller.
  2. Send a StopCanister message from the controlling canister.
  3. Lower the threshold and remove the canister controller.
5 Likes