DFX Warning when running locallly - Invariants check failed with err: Invariant broken:

Developers
1

@claudio @chenyan

I just started playing around locally with performing inter-canister calls in parallel from a canister to the management canister, and ran into the following error.

(This error is one long line, but I’m inserting newline characters to make it easier to read)

May 23 05:21:29.136 
WARN s:utaii-kuahc-dtn2h-q52ef-kl77v-yozyo-pppml-dxhtm-u6zo5-ifzxe-jae/
n:lihjb-4jgo7-q3rph-633jm-sfszg-g2sov-6ppfr-u2r5e-rxhi2-ewgts-7qe/
ic_execution_environment/scheduler 
At Round 170526 @ time 2022-05-23 05:21:28.403547 UTC, 
canister t6rzw-2iaaa-aaaaa-aaama-cai has invalid state after execution. 
Invariants check failed with err: 
Invariant broken: Canister t6rzw-2iaaa-aaaaa-aaama-cai: 
Number of call contexts (3) is different than the accumulated number 
of reservations and responses (4), 
messaging: {"round":170526,"canister_id":null,"message_id":null}

My setup is the following:

I have one main canister, and two child canisters. If I attempt to stop both child canisters in parallel, the call succeeds and stops both canisters, but I receive this WARN + invalid state after execution error which makes me worry about any side effects.

For reference, the main canister is the t6rzw-2iaaa-aaaaa-aaama-cai complaining about inconsistent state that you are seeing in the error message.

Here is the jist of what I’m doing to execute the stop canister requests in parallel from the main canister.

To perform parallel execution, the main canister calls the code below and is first executing each of the asynchronous calls in into the executingStopCanisters buffer, and then I’m collecting/awaiting the results of those asynchronous calls in the next round into the collectingStoppedCanisters buffer.

The CA.stopCanister method is just a refactored version of the stop_canister method

  public func stopAllCanisters(canisterPrincipals: [Principal]): async [Text] {
    let executingStopCanisters = Buffer.init<async Text>();
    for (principal in canisterPrincipals.vals()) {
      Buffer.add<async Text>(executingStopCanisters, CA.stopCanister(principal));
    };
    let collectingStoppedCanisters = Buffer.init<Text>();
    var i = 0;
    label l loop {
      if (i >= executingStopCanisters.count) break l;
      Buffer.add(collectingStoppedCanisters, await executingStopCanisters.elems[i]);
      i += 1;
    };
    Debug.print("all stops complete"); 

    Buffer.toArray(collectingStoppedCanisters);
  };

Is this a serious error? Should I be concerned about any side effects to the main canister making these calls?

2

These messages about call contexts can be safely ignored. There was a bug in the state invariant check that was fixed in Merge branch 'exc-1035' into 'master' · dfinity/ic@d8dd2d1 · GitHub
Note that the fixed message would say “Number of callbacks … is different than the accumulated number of reservations and responses …”

Looks like DFX got the version of the replica that has the bug, but not the fix.
Besides printing this bogus messages the bug doesn’t have any impact.

2 Likes
3

Thanks for the response and assurances, I upgraded my dfx from 0.9.3 to 0.10.0 and this error looks like this bug is in that version as well. Will a fix for this be included with the next release of dfx?

4

We always include the latest blessed replica release when we make a new dfx release, so yes, this will be fixed in the next version of dfx.

1 Like