And when running this REPL with 200 as an argument, the following error appears.
Server returned an error:
Code: 400 ()
Body: Specified ingress_expiry not within expected range:
Minimum allowed expiry: 2022-10-21 15:32:10.019462710 UTC
Maximum allowed expiry: 2022-10-21 15:37:40.019462710 UTC
Provided expiry: 2022-10-21 15:32:08.119 UTC
Local replica time: 2022-10-21 15:32:10.019464171 UTC
It looks as if the agent signs the signature request for the state read (i.e. for polling the response) when it makes the call, and when the call takes a while (e.g. a loop around await) does not extend the expiry in the read request, and after a while it expires.
The problem is that when 100 users call the canister at the same time, the number of Output Message Queues may exceed the limit of 500, and some await calls may be trapped.
My views:
The await call is classified as outcall and innercall, and the innercall should not be so restrictive.
Optimise the await role of private function calls. As in the above example, it is sufficient to make await ledger.transfer(…) act asynchronously, ignore other await. An effect like this.
There is a new solution: add the countAwaitingCalls() method inside the ExperimentalInternetComputer Module so that Canister can limit the entry of new requests.
In your original example, each call uses await, which means it will schedule an outgoing self call message, and end the current call. So the total number of outstanding messages do not increase.
It is only when you use ignore fun2() for example, it will schedule more than one outgoing calls.
Edit: I should add that calls like await fun2() will also reserve resources (e.g. place in the input queue) to make sure when fun2() returns a value, it will be processed. So nested await calls do consume more resources than a single one.
Yes, Output messages are accumulated whenever an ignore funN() is present in the call chain. This becomes uncontrollable when many users access it at the same time.
I talked to some people today to collect some recommendations on how to handle issues with too many outstanding messages filling up queues. Two general recommendations to be followed when aiming for scalable dapps that call other canisters came up quite consistently:
Make sure that the dapp maintains a counter or something similar on how many outstanding requests it has, and explicitly handle the situation where too many calls would be in flight at the same time. @roman-kashitsyn agreed to follow up with details how this is (plannded to be) done for the ckBTC canister.
If the design of the dapp allows to batch together some of the calls to external canisters, aim to batch them together. For example, if there are multiple calls to the ledger involving the same account, it might be possible to batch them together and only do one transfer.
There are also certain things that the IC protocol could do differently. The things identified here seem to be in line with the suggestions already brought up earlier in this thread. However, I want to stress that these measures will not really help with scalability as these would only bump limits by an order of magnitude or even less, but limits would still be easily hit as soon as, say, 1000 instead of 100 users try to do something. These things are:
Investigate whether it can be made easier to make nested function calls in Motoko without accumulating reservations, or whether there is an alternative pattern one could use. @PaulLiu already provided some pointers above, and @claudio agreed to follow up on details on this and what could be done.
Calls to self are currently treated in the same way as calls to other canisters. This means that there is a reservation for the response made in the input queue and in the output queue, which means that for calls to self effectively only have half the queue capacity available. This item is already in our list of backlog tasks and we will look into whether this can be picked up soon.
Yes, this point is what @claudio will provide more details on. This is what I aimed to describe in the bullet point “Investigate whether …”.
What I meant to describe with the bullet point on reservations for responses in the snippet you are citing in your previous message is something that could be improved on the protocol level thats unrelated to how these things are handled in Motoko: roughly speaking, the protocol currently only allows DEFAULT_QUEUE_CAPACITY/2 requests in flight to self while there can be DEFAULT_QUEUE_CAPACITY requests in flight to other canisters. This is because the protocol doesn’t distinguish between local and remote canisters; distinguishing between them could provide 2x more space for messages to self.
I will write a longer response when I get a chance, but, for now, to avoid the overhead of async/await associated with local functions that need to send messages, you need to remove those functions and inline them into their call sites.
I agree this is not good and have even proposed and implemented solutions to this problem in the past but they were felt to be too risky, blurring the distinction between await and state commit points.
I’ll elaborate on this in another reply, but fully agree that the current situation is not good enough for code-reuse and abstraction.
I’m happy to revisit addressing this, but there is no quick fix beyond inlining the calls to avoid the redundant async/await.
This is important. motoko is not a toy, not just for writing demos. motoko needs to meet the needs of engineering.
Its risks can be improved by good programming habits, good IDE tools.
Calls to functions of smart contracts in EVM are also divided into internal and external calls.
Yes
The introduction of new semantic expressions is a good solution. For example inner await, inner async. inner await is not a data commit point, but it may have an await data commit point inside it.
Indeed, FTX Storms, the centralization is facing more and more challenges in the foreseeable future and we need to be prepared for these users who are moving to decentralization!
Looking forward to detailed reports and what preparations and changes we need to make(as soon as possible!) to support tens of millions(even more,yes even more) of users!
“Canister trapped explicitly: could not perform self call" error caused by input/output message queue limitation cannot be caught by try-catch now.
This can break data consistency.
For example
private stable var n: Nat = 0;
private func fun() : async (){
try{
n += 1; // Here it has been executed
let res = await fun1(); // trapped by input/output message queue limitation
// ... // Here the code will not be executed
}catch(e){
n -= 1; // Here the code will not be executed
};
};
This is expected behavior - if the error comes at runtime from the canister itself then (in this case the canister is overflowing it’s own output queue), then the error can’t be caught and traps. Until nested async function calls are optimized or the canister output queue limit is raised, I would recommend putting in guardrails in your code to protect you from these runtime trap situations.
If the error comes as the awaited response from an async call, then this can be caught. Likewise, errors explicitly thrown by your code in a canister can be caught.
We discussed the async abstraction issue in our team meeting today and will prioritize finding a solution for this soon.
But it won’t be overnight, I’m afraid, so you’ll need a workaround for now.
One is to avoid using asynchronous local functions by inlining their bodies into the call-sites, without the awaits.
If you don’t like inlinng, another solution that might work is to write non-async functions returning a value describing the call you want to make (a tuple of shared function and arguments), and getting the outermost function to perform the actual call with a single await, by applying the function from the tuple to the argument in the tuple and awaiting that.
Now our temporary solution is to actively control the number of input/output messages. This control is not very reliable because it is difficult to get the exact size of the message queue.