The protocol doesn’t provide any guarantee that a destination canister 100% processes a request. The only guarantee the protocol gives is that every request will get exactly one reply. However this reply can also be system generated in case the request can not be delivered/processed (OOM, queue full on the receiver side, destination canister trapping… etc.). So what you can rely on is that you will eventually receive a reply that will tell you what happened to the request.
This is also true for state 1. If a request times out in the output queue the canister will get a system generated reply that the message timed out. So this statement is also not correct:
To sum up: it doesn’t really make sense to distinguish between the two states you sketch above from a canister’s perspective as there are no delivery guarantees in any of these cases. In both states the only thing you can rely on is that you will get a reply which is gonna tell you what happened to the request. Based on this one can then make a decision on whether or not to retry.
Since we’re working with Notify here, the canister that issues a Notify call doesn’t even get that, correct? The only thing we know from the sender canister is that the Notify was successfully added to the outgoing queue or not (the Notify call can either succeed or fail, in sync mode).
Then there’s the question of how often shall we re-try a call that wasn’t answered (via another Notify from the other canister). My intuition here is that our lib should implement something like retry with back-off. First retry at x seconds, then x*2, etc. for a number of retries and then just give up on the call.
Since we’re generating a unique ID for each new flow, we should be OK even if sometimes 2 messages reach a canister. The lib should cover the case where an identical message_id was received, and not re-process the message, instead re-issue the ACK (as described by Austin in the main bounty proposal).
I’m not familiar with notify but I assume that notify will just pass invalid callback IDs when making the calls, right?
If so, keep in mind that notify will make the canister not see the reply to a notify but this doesn’t mean that there is no reply. The system still makes a reservation for the reply and the notify will consume a slot in the queue until that reply (which may be system generated if the notified canister doesn’t reply explicitly) arrives. It is just that when consuming the reply the invalid callback ID will make sure that no canister state is changed. So if you retry too aggressively you will end up filling up your own queue and enqueuing new requests/notifys will eventually fail.
This is what I was thinking. I think the question is that with the 5 minute queue, should the first retry happen before or after the 5 minutes(give or take some time).
I realized here that there is a flaw in the com_asyncFlow_fin (or com_asyncFlow_ackack) function Since it is sent once (without resending attempts), I think this flaw will need to be fixed. But that’s not the point. I realized here that the sending attempts themselves (if there is no confirmation) load the network. Let’s say 10 attempts to send a new message and 10 confirmation attempts load the network (10+10)/2 = 10 times!. The two here is the count of the usual exchange. The way out of the situation I see so far is this: the 1st attempt is single, perhaps the second attempt is single. Further confirmation attempts( let’s say from the 3rd to the 10th) must be sent in bulk (i.e., using Data Collections (List) ). That is, send multiple confirmations in one request
To my understanding, the purpose of this library is to allow canisters to communicate with 3’rd party canisters without worrying about malicious actors that can render your own canister un-upgrade-able. For context, check out this post. It is meant as a stopgap until a permanent solution to the upgrade issue is implemented by Dfinity.
Using this library will be ~3x more expensive, but it will enable untrusted 3rd party communication today. Some people might find the tradeoff worth-while.
I understand. I’m also leaning towards the main option. But there is an option to optimize the number of requests. I think after writing the main library, it will be possible to think about improvements.
I have a ready-made implementation. She was ready a couple of days ago. But there are packaging problems. Link to github
Now about the problem: the actual asynchronous data exchange in this library excludes the use of async — await operators. Using these operators will result in waiting for the result to be returned. My code in the implementation of actors (Sender — Receiver) does not use them. Since they are immediately created in canisters during assembly.
Of course, the main task of the library is to hide (encapsulate) the work of the library and simplify its use. Therefore, I wanted to take out the main logic of the work, that is, the code into a separate class or module. But in this case, the compiler (SDK) requires the use of asynchronous functions.
For example, a section of code in the sender’s canister (not packaged):
But if this code is packaged in a class or a separate module. Then it turns out such a construction:
(incorrectly)
class SourceSender(){
public shared({caller}) func com_asyncFlow_newMessage(msg: MessageType): async(){
//****//
await canister_receiver.com_asyncFlow_newMessage(msg);//NEW
}
}
I will think about how to get away from asynchrony in the class. Perhaps the callback functions will help in this. Update 1 (the callback function also requires asynchrony in the function parameters).
What are your opinions on this?
I think this requirement is expected. You may want to use async* in your library so that if the library cancels the send for some reason it doesn’t actually cause an await.
That’s right, I’m not using async-await* at the moment. Everything works fine without async-await. But this is in Sender-Receiver cans, but if the code is packaged in a separate module, the SDK (compiler) forces you to add code with async-await*, otherwise calls between containers are not compiled.
I’m still considering exit solutions out of the problem. I thought the callback functions would help, but they also eventually require the async-await construct.*
I also realized that the created class (in the actor) and even the module (in the actor) will be interpreted by the scanner and/or compiler as containers and require an asynchronous async-await* construct.
Update 1 I think I’ve found a solution, but I’m not 100% sure
I’ve been using your recommendations here. They were needed, thanks again.
I also have a question about asynchrony.
Looking through the forum thread and sdk:
I found some innovations on async* and await*, but they require the return of the result and not void. It is also not known until the end whether it will work correctly. This is more useful for internal calls within a single canister. And the async-await(async*-await*) construction introduced once is then required everywhere.
I would like to leave two solutions:
This is a template (example) of two canisters. Where there is no asynchrony.
Library. Where forced asynchrony is used. Asynchrony will be used until the situation is clarified or in connection with new developments in the language.
//version in actors
actor S{
func a(){
canister.send();
}
}
//version wrapped in a library
actor Other{
let lib = Lib()
func a():async(){
await lib.a();
}
}
module{
class Lib{
func a():async(){
await canister.send();
}
}
}
The first option is “sent- forgot”
The second option is waiting for the result.
Although in both cases there will be an answer either with an error or with a result.
It would be ideal to do following the logic of “sent- forgot”. But it doesn’t work! (If wrapped in a class) Therefore, at the moment I have the library wrapped in classes with an asynchronous version.
I think async* await* is more preferable. I’m not 100% sure, but this should throw off the expectations of in-module calls within a single container.
In truth, I have formed the opinion that a normal call is not much different from an asynchronous one within this SDK. Since in any case there will be a response either from the system or from the canister.
I had a version with async* await*, then I removed it. Since there are modular calls inside and calls between canisters within the same function. In my case, I am not sure of the correctness of such code. In a simple version, where there is one call and it is inside one module, this should work.
I’m in the final stages now. I need to arrange and pack in mops. I think I’ll come back to the async-await issue later.
Hi!
I have done some improvement and optimization work.
I will describe it more briefly:
-System timers have been moved to the library. This is now hidden from library users.
-A stable version of hashmap has been added, in theory it should be faster than triemap.
-Changed the use of asynchrony (async* await*)
-Now at the first unsuccessful attempts to send and confirm. Repeated attempts are sent in packets satisfying the condition (by time)
Hi!
I haven’t seen what could be improved yet.
The only thing that was done: this is a complete refactoring of the code.
I also want to add one more detail. It concerns the synchronic of async* await*. There is no support at the moment in timers timer.recording Timer(time, job)
(synchrony with an asterisk)
It’s not critical. I asked this question. (System Timer support async*)
But there is no support in the latest version (MOC) yet.