LAMENT: A tale of constant struggle of what it's like trying to scale on ICP

My guess is that you do this work with a timer over many rounds? And that round may call a canister on the same/or simally contested subnet to get some value or distribute data? If you don’t increase your compute allocation on all of the involved canisters, then yes, you will see this behavior.

Example:
Canister A sets timer for next round
Canister A is not scheduled for 23 minute.
23 minutes later: Canister A activated and does work but awaits canister B for a value to finish work.
Canister B is not scheduled for up to 23 minutes.
Up to 23 minutes later(+46): Canister B processes and returns value to A
Up to 23 minutes later(+69), Canister A is scheduled to process the request and does loop 2.

If you do a few of these loops you end up with a very long wait.

If this work can be parallelized at all you should do so. Initiate all the calls with a future and then await them all at the same time. Make sure you have enough cycle to cover all the potential calls(up to 500).

I’m guessing your work is minting or sending tokens to a group of accounts? Parallelize it.

See this thread: All or nothing batch transaction ICRC standard? - #14 by icme

1 Like