I suppose you mean “when queue is full”?
However, I think when await bbb() traps, since it is treated as a synchronous error, the execution of function aaa() will roll back and counter shouldn’t be incremented.
So what @bitbruce wrote above looks correct code to me. No?
Right on both counts but he still needs to test counter to avoid the trap in the first place.
So is the main issue being discussed if you chain more than 100 some ignore’s() of any asynchronous call to another canister? I’m trying to understand whats the issue from a motoko only perspective, if you will.
Now I also noticed you mentioned Language Support - Internet Computer Developer Forum this post where you describe some kind of parallel execution which I havent fully wrapped my head around tbh. But essentially I do see that the stable buffer add is calling an async a bunch of times without having to await anything. Is under the hood, this function just ignoring instead of awaiting each f(as[i]), and at the end of the day and its the same problem? Or is this something completely different? Or is this not even where the error is in that code sample?
I think @claudio meant the code inside bbb() traps. Shouldn’t then await bbb() cause an error with reject code 5, that get caught and inside the catch branch the counter gets decremented again? So we don’t roll back. The counter value is correct, just for a different reason. So the code from @bitbruce still looks correct to me. If it does not work then I would like to understand why it doesn’t work.
Actually, wait when @PaulLiu says
Does he mean
or is he saying
Meanwhile youre saying
I think the last one is correct though.
No I actually meant that the call to bbb() traps in aaa()before ever entering bbb() (because the queue between the actor and the destination of bbb is full). Then aaa() traps and all its effects, including the increment of counter, are rolled back.
We could make the trap (before entering bbb()) produce a local exception in aaa(), transferring control to the catch clause, but the current implementation does not do that.
I’m about to investigate changing that.
Ok, but in that case Paul’s reasoning applies, the increment will be rolled back, and the code is correct. I still don’t understand how the code can break as observed in the stress test.
A few follow ups that just came to mind.
What about adding this API with a limit to the # of principals passed (say 1000 canister principals)? That’s ~33KB.
It’s a huge improvement from a cycles cost perspective, since then only one inter-canister call is required.
Is there a particular reason why this queue capacity limit is currently 500 and not higher (i.e. performance degradation or DDOS prevention?)?
What about adding this API with a limit to the # of principals passed (say 1000 canister principals)? That’s ~33KB.
This would alleviate the issue of the result potentially being too large. But I think we would still need to handle what @bogwar mentioned
Further to @dsarlis 's point regarding
canister_status: these calls are passed to the subnetwork hosting the canister so batchingcanister_statuswould not work w/o a lot of abstraction breaking and low level machinery and bookkeeping (i.e. parsing the message, constructing messages for the different subnetworks and then reconstructing the reply from the replies we would get)
Making this work is non-trivial. We could also say that we can limit this api to only canisters on the same subnet but at this point I’d be questioning whether that API would be really useful.
Is there a particular reason why this queue capacity limit is currently 500 and not higher (i.e. performance degradation or DDOS prevention?)?
I believe the main reason is to ensure that no single queue can potentially take all space, so more like DDOS prevention as you mentioned. A limit has to be there imo, we cannot have unbounded queues. Whether the 500 value is too low or not, I’m not sure I can answer off the top of my head. We’d likely need to experiment with higher values and see if it would be viable from a subnet health pov. Is there a specific value that would be more preferable? 2x? 3x?
I see how this would break the existing endpoint interface and require a new endpoint or type definition, but I’m not sure I quite understand the heavy lifting involved in sending out multiple parallel asynchronous requests from the subnet and then collecting the response - isn’t the subnet good at handling this type of stuff when dealing with cross-subnet calls?
In order of preference, being able to batch principals to canister status would be preferred as that would allow me to drastically reduce the canister output queue between my status checking canister and the IC management canister while simultaneously reducing cycle costs.
If this isn’t possible 2X would be a good start for the output queue being raised, but I’d just be throwing more parallel requests at the management canister.
I probably don’t fully understand the plumbing involved in what @bogwar and you are referring to, but it feels like having a batch canister status method would actually reduce load on the management canister, as fewer status calls would be necessary.
The same goes for many canister management methods.
in sending out multiple parallel asynchronous requests from the subnet and then collecting the response - isn’t the subnet good at handling this type of stuff when dealing with cross-subnet calls?
What you’re asking is an endpoint which when you call it, it would fan out to multiple messages and then collect the results. This would likely require keeping some call context on the management canister for these requests as well as some logic to be able to know which messages for other subnets we need to send (based on target principals). It’s not that we can’t do it but it would require quite some work. Also, where things get a bit hairy is that normally routing messages to other subnets happens on the message routing layer but in this case the management canister would probably need to do some of it (hence Bogdan’s comment on abstraction breaking).
As I said earlier in the thread, it’s a nice idea but implementing it might be trickier than you initially think. I can bring it up with some folks internally and see if we can sketch out something to gauge the effort level more concretely.
Caution is indeed needed if the solution to the issue is too costly.
A more conservative approach would be to
Was further looking into queues, and found DEFAULT_OUTPUT_REQUEST_LIFETIME
There’s also reference to a deadline and requests “timing out”
Is there a certain point at which inter-canister messages could be dropped (timed out) if the deadline has passed? Let’s say I have a canister or subnet that’s completely overloaded with messages and has a bunch of these messages in my canister’s output queue. What happens to these messages if the 300 second deadline has passed?
Note that the protocol guarantees that each request will receive a response. So requests are never dropped. Instead the system produces synthetic reject responses whenever a request can not be delivered/processed.
More precisely, for for timeouts this means that all requests that don’t make it into the subnet-to-subnet stream before the deadline has passed (and therefore were not visible to anybody except the sending canister), the system will produce a synthetic reject response in the same way as it would produce a synthetic reject response when, e.g., the receiver’s input queue is full, the receiving canister/subnet is at memory limits, the receiver is stopped/freezed/traps while processing, and potentially other cases I might be missing now. So everything can be handled in the same way as in these cases and canisters shouldn’t be required to explicitly handle these timeout cases – that is, assuming they handle the other cases correctly, they already implicitly handle timeouts, as a timeout is also just an asynchronous reject with reject code “transient”. Requests in streams currently don’t time out as this would require more work to implement in a way that messaging guarantees given by the IC are not broken (essentially once messages are in streams we don’t know whether the message has been seen/picked up by other subnets/canisters and therefore no longer have an easy way to time them out). Finally, for responses, the protocol guarantees delivery and so these won’t time out at all, irrespective of whether they are in streams or in queues.
Hi @bitbruce, the changes that were necessary to also allow calls to self to go to the full queue capacity have meanwhile been made and are part of the release elected to be rolled out this week. Hope this helps to make things smoother for you and other devs.
Great work!
We will be working on the canister code upgrade.
For those following this topic thread,
Tomorrow @dsarlis will be giving a talk tomorrow in the Scalability & Performance working group on “Canister queues (ingress, input and output) overview and “case studies of handling large message volumes” - Technical Working Group: Scalability & Performance - #33 by domwoe
Motoko 0.8.1 is out now. The previous release, 0.8.0, introduced the following (breaking) change to let users detect and handle message send failures using try ... catch expressions. These releases are not yet included in dfx to allow time for users to provide feedback.
BREAKING CHANGE
Failure to send a message no longer traps but, instead, throws a catchable Error with new error code #call_error (#3630).
On the IC, the act of making a call to a canister function can fail, so that the call cannot (and will not be) performed.
This can happen due to a lack of canister resources, typically because the local message queue for the destination canister is full,
or because performing the call would reduce the current cycle balance of the calling canister to a level below its freezing threshold.
Such call failures are now reported by throwing an Error with new ErrorCode #call_error { err_code = n },
where n is the non-zero err_code value returned by the IC.
Like other errors, call errors can be caught and handled using try ... catch ... expressions, if desired.
The constructs that now throw call errors, instead of trapping as with previous version of Motoko are:
shared functions (including oneway functions that return ()).async. These involve sending a message to self, and can fail when the local queue for sends to self is full.async expressions. These involve sending a message to self, and can fail when the local queue for sends to self is full.await expressions. These can fail on awaiting an already completed future, which requires sending a message to self to suspend and commit state.(On the other hand, async* (being delayed) cannot throw, and evaluating await* will at most propagate an error from its argument but not, in itself, throw.)
Note that exiting a function call via an uncaught throw, rather than a trap, will commit any state changes and currently queued messages.
The previous behaviour of trapping would, instead, discard, such changes.
To appreciate the change in semantics, consider the following example:
actor {
var count = 0;
public func inc() : async () {
count += 1;
};
public func repeat() : async () {
loop {
ignore inc();
}
};
public func repeatUntil() : async () {
try {
loop {
ignore inc();
}
} catch (e) {
}
};
}
In previous releases of Motoko, calling repeat() and repeatUntil() would trap, leaving count at 0, because
each infinite loop would eventually exhaust the message queue and issue a trap, rolling back the effects of each call.
With this release of Motoko, calling repeat() will enqueue several inc() messages (around 500), then throw an Error
and exit with the error result, incrementing the count several times (asynchronously).
Calling repeatUntil() will also enqueue several inc() messages (around 500) but the error is caught so the call returns,
still incrementing count several times (asynchronously).
The previous semantics of trapping on call errors can be enabled with compiler option --trap-on-call-error, if desired,
or selectively emulated by forcing a trap (e.g. assert false) when an error is caught.
For example,
public func allOrNothing() : async () {
try {
loop {
ignore inc();
}
} catch (e) {
assert false; // trap!
}
};
Calling allOrNothing() will not send any messages: the loop exits with an error on queue full,
the error is caught, but assert false traps so all queued inc() messages are aborted.
Hey Claudio, to clarify - this breaking change is only changing how output queue overflow errors will be handled, correct? (They need to be caught and explicitly trapped if the developer wishes to discard the intermediate state and not have it committed).
My understanding is that other inter-canister or non-output queue async/await errors already demonstrate the intermediate state commit behavior.
…Or is this a complete overhaul of how all async errors are handled?
Correct.
If a call itself returns an error (by rejecting the message or trapping) the caller’s state will, by necessity, already have been committed (otherwise the call would never have been issued).