I might be missing it, but I can’t find where the behavior of a canisters mailbox is described.
Couple of questions:
Hi Hazel!
Can an actors mailbox “fill up” and reject new messages?
Between each pair of canisters, there is a queue of outstanding messages. If this queue is full, then canisters will not be allowed to create new messages.
Note that the IC has guaranteed responses so we want to make sure that when a canister is trying to produce a response, it never hits the full queue issue. Hence, when a canister tries to produce a request, we reserve a slot for the eventual response. So a canister should only experience the full queue error when trying to produce a request.
- Is it possible to upgrade a canister while it still has messages to process? If so, do those messages remain enqueued or is the queue purged?
A canister can always be upgraded regardless of how many messages it has in its input queues. However, in practice, it is generally hard to write applications that one can upgrade and still be able to make sense of old messages. More specifically, if a canister sends a request to another canister, it may create some state locally to help process the eventual response. If the canister is upgraded before the response comes back, making sense of the state might be a difficult problem to solve.
Hence, we have the feature to “stop” a canister. We generally recommend that the controller of a canister should first stop it and only try to upgrade a stopped canister. When you stop a canister, it stops processing new requests (just responses) and when it has processed all responses, it becomes stopped.
I hope this helps.
Thanks for your response! Two questions after sitting on this for a day:
- Did I understand this correctly? So, for a bucket B and two caller C1 C2 → (C1 calling B) and (C2 calling B). Both C1 and C2 would have their own queue limits between B?
Yes. B has two distinct queues, one for C1 and for C2, each with their own limits. So you should not have the problem of C1 using up all the queue space for B preventing C2 from getting its messages through.
Some other related notes though:
- Is this behavior the same from the context of an ingress message? Anon Principal / Self-Identifying.
Each canister currently has a single queue for all ingress messages. So if you U1 produces a ton of messages for a canister and then U2 produces a single message, U2 will have to wait for all of the messages from U1 to processed before it gets a chance to run.
Another property of the ingress queue is that it is technically an infinite queue as in you can always push more messages on it. However, because ingress messages have an expiry time and due to the block rate limit of the subnet, there is an upper limit on how many ingress messages can be produced per minute, we do have an upper bound on how many ingress messages can be in a given queue.
As you can see, there is room for improvement all over here. We hope to be able to make the desired improvements in iterations.
Hello, thank you for your detailed answer, but I still have two questions about the canister mailbox.
Based on my understanding of Akhilesh’s response:
- According to the answer about actor mailbox as above, a canister actually have many queues between canister and canister, canister and users.Is that right ?
Yes, a single canister has multiple queues. One for each canister that calls it (i.e. inter-canister calls), but a single shared queue for all users that call it (i.e. ingress calls).
- The queues of canister and canister use the RR scheduling, but how about the canister and users messages?
All ingress messages (produced by users) go to a single queue for a given canister. So round-robin doesn’t make sense here, since there’s only one queue. It’s just regular FIFO.
Thanks @jzxchiang ! That reply’s spot on!
This is the corresponding technical talk on this. Relevant part here
Though I would advise watching the whole thing 