Retired: ICDevs.org Bounty #38 - Async Flow - One Shot - Rust - $6,000

[skilesare]

Async Flow - One Shot - Rust - #38

Current Status: Discussion

• Discussion (01/09/2023)
• Ratification: (01/09/2023)
• Open for application: (01/09/2023)
• Assigned
• In Review
• Closed

Official Link

Bounty Details

• Bounty Amount: $6,000 USD of ICP at award date.
• ICDevs.org Bounty Acceleration: For each 1 ICP sent to 66083bb3c940c7de2a70aa127ec771da75d60deadc10f5f66b280f203d8f6e73, ICDevs.org will add .25 ICP to this issue and .75 ICP to fund other ICDevs.org initiatives.
• Project Type: Individual
• Opened: 01/09/2023
• Time Commitment: Days
• Project Type: Library
• Experience Type: Intermediate - Rust;

Description

The IC implements an asynchronous messaging system where requests are made to canisters and a transaction id is returned. The canister then queries the state of this transaction and returns the result when it detects that the function is complete. The Rust CDK and Motoko abstract this away from the user in a way that depends on the IC reliably fulfilling the request with certain guarantees.

Sometimes a canister developer may want to do away with this abstraction and implement their own async flow when the results of the called function are not important to the continuation of their code. This is more event based programming and it is especially useful while the IC still requires functions to return before upgrades can be performed. Future functionality will fix this upgrade issue, but async and event based programming is still a useful pattern when services are interacting. It removes dependencies and allows the developer to slip into an actor based frame of mind that more closely mirrors how the IC is actually working under the covers. Specifically, it can keep the developer from making “await” assumptions that open the canister to reentrance attacks.

In this bounty the user will create a library to handle asynchronous messaging.

When a user initiates an async one-shot call they likely do want to handle some kind of response so that they can confirm that the call was received. In turn, the acknowledger needs to know that the acknowledgment was received. You end up with something that looks a lot like a TCP/IP flow.

This pattern should expose a number of functions from the canister to implement this flow:

Sending Canister:

com_asyncFlow_ack({msg_id: nat; hash: ?nat32; payload: blob}) -> () //confirm that a message was received. payload is candid encoded data

Receiving Canister:

com_asyncFlow_newMessage({msg_id: nat; payload: blob;}) -> () // process a new message - payload is candid encoded data
com_asyncFlow_ackack({msg_id: nat; hash: ?nat32}) -> () //acknowledges that the acknowledgement was received

The library function signature should look something like:

constructor :  ( return_async: (canister_id: principal, sent_payload : blob, received_payload : blob, msg_id: nat, hash: ?nat32) -> async* ()); //this function is called when an ack is recieved.
call_async(canister_id: principal, payload: blob) -> async* nat; // the nat should a ULID encoded as a Nat : https://github.com/aviate-labs/ulid.mo/blob/main/src/ULID.mo

Since the IC has a new 5 minute timeout on full queues, the Sender and Receiver should retry this messaging after 5.5 minutes. Therefore the library needs to keep state of sent messages and should discard those messages after the acknowledgment has been made.

The library should also keep track of processed message ids such that if a duplicate message is received, it is not reprocessed and instead, an acknowledgment is sent.

The hash is a check to make sure the acknowledgment was of the correct data. If an improper hash is received the library should hand the request to a corruption handler with enough data to manage the error.

This bounty gives the opportunity to

• learn about rust
• learn about async messaging
• learn about state management in rust

To apply for this bounty you should:

• Include links to previous work writing tutorials and any other open-source contributions(ie. your github).
• Include a brief overview of how you will complete the task. This can include things like which dependencies you will use, how you will make it self-contained, the sacrifices you would have to make to achieve that, or how you will make it simple. Anything that can convince us you are taking a thoughtful and expert approach to this design.
• Give an estimated timeline on completing the task.
• Post your application text to the Bounty Thread

Selection Process

The ICDevs.org developer’s advisors will propose a vote to award the bounty and the Developer Advisors will vote.

Bounty Completion

Please keep your ongoing code in a public repository(fork or branch is ok). Please provide regular (at least weekly) updates. Code commits count as updates if you link to your branch/fork from the bounty thread. We just need to be able to see that you are making progress.

The balance of the bounty will be paid out at completion.

Once you have finished, please alert the dev forum thread that you have completed work and where we can find that work. We will review and award the bounty reward if the terms have been met. If there is any coordination work(like a pull request) or additional documentation needed we will inform you of what is needed before we can award the reward.

Bounty Abandonment and Re-awarding

If you cease work on the bounty for a prolonged(at the Developer Advisory Board’s discretion) or if the quality of work degrades to the point that we think someone else should be working on the bounty we may re-award it. We will be transparent about this and try to work with you to push through and complete the project, but sometimes, it may be necessary to move on or to augment your contribution with another resource which would result in a split bounty.

Funding

The bounty was generously funded by the DFINITY Foundation. If you would like to turbocharge this bounty you can seed additional donations of ICP to 66083bb3c940c7de2a70aa127ec771da75d60deadc10f5f66b280f203d8f6e73. ICDevs will match the bounty $40:1 ICP for the first 75 ICP out of the DFINITY grant and then 0.25:1. All donations will be tax deductible for US Citizens and Corporations. If you send a donation and need a donation receipt, please email the hash of your donation transaction, physical address, and name to donations@icdevs.org. More information about how you can contribute can be found at our donations page.

FYI: General Bounty Process

Discussion

The draft bounty is posted to the DFINITY developer’s forum for discussion

Ratification: (01/09/2023)

The developer advisor’s board will propose a bounty be ratified and a vote will take place to ratify the bounty. Until a bounty is ratified by the Dev it hasn’t been officially adopted. Please take this into consideration if you are considering starting early.

Open for application

Developers can submit applications to the Dev Forum post. The council will consider these as they come in and propose a vote to award the bounty to one of the applicants. If you would like to apply anonymously you can send an email to austin at icdevs dot org or sending a PM on the dev forum.

Assigned

A developer is currently working on this bounty, you are free to contribute, but any splitting of the award will need to be discussed with the currently assigned developer.

In Review

The Dev Council is reviewing the submission

Awarded

The award has been given and the bounty is closed.

Other ICDevs.org Bounties

[GLdev]

Hey, I might take this on, since I was discussing with @domwoe about writing an article on state handling with enums in rust. Handling state over many messages might be the perfect example for such an article.

Couple of quick questions / observations:

1. are the hashes really needed? I’ve only briefly looked at the one way messages, but AFAIK they have all the guarantees of “regular” messages on the IC, with the exception of handling the response. That is to say, if you get a message, it is guaranteed to be the message that the sender intended. Computing a hash would only lead to wasted cycles, IMO (if my assumption on the message contents guarantees is correct).

Keeping the first msg_id around would be enough in this case.

2. com_asyncFlow_ackack could probably be renamed to something like com_asyncFlow_fin as ackack makes me think of green little martians pew pewing in the streets

@skilesare

The library should also keep track of processed message ids such that if a duplicate message is received, it is not reprocessed and instead, an acknowledgment is sent.

Hmmm. From the description (and prior conventions) the ACK message is simply an acknowledgement of a message receipt, it says nothing about if or how the message was processed on the receiving end. The receiving canister would need to implement and track a variety of intermediary states if it needs to avoid reprocessing (status_processing, status_completed (return_value), status_fin) and implement functions to communicate these state changes.

In the UDP standard (something that resembles the one shot messaging on the IC) this is left as an Application layer, as this heavily depends on the business needs. Having a library take care of these implementation details would serve little purpose, IMO. Some apps might need this, while other might not, and a simple ACK would suffice. And for the apps that need it, they might want to track and handle responses, errors, complicated flows, etc.

@skilesare

Since the IC has a new 5 minute timeout on full queues, the Sender and Receiver should retry this messaging after 5.5 minutes. Therefore the library needs to keep state of sent messages and should discard those messages after the acknowledgment has been made.

Same as above, the retry logic might heavily involve business logic so abstracting it at this point might not be the best.

Can we perhaps come up with a couple of real-world examples (say a trivial one - logging? and a more complicated one with a more involved flow) and see how that goes? At the very least we can point people towards some example apps for a basic implementation, see how it is received and then perhaps group some functionality under a library?

[skilesare]

@GLdev

are the hashes really needed?

I made them optional. I think the answer is that I’m not sure. Ideally, an app that received a response with a message ID would confirm that the ID exists. Theoretically, a boundary node could inspect the message and send back a fake request! They’d have to simulate the principle id, but the hash is there if you want to make sure the receiver actually got the message. Although I guess if the boundary node saw it they would be able to simulate it. You’re free to come up with some best practices and theories about how to keep it secure. I would like the motoko and rust implementations to match.

@GLdev

com_asyncFlow_fin

Yes…probably better.

@GLdev

In the UDP standard (something that resembles the one shot messaging on the IC) this is left as an Application layer, as this heavily depends on the business needs. Having a library take care of these implementation details would serve little purpose, IMO. Some apps might need this, while other might not, and a simple ACK would suffice. And for the apps that need it, they might want to track and handle responses, errors, complicated flows, etc.

I think if you can lay out the best practices of how to handle this then that would suffice. I think I was thinking of a scenario where the Message was received by the receiver, but then the ack failed…maybe because the outgoing queue was full with the new 5 minute timeout thing. In that case, the source will likely send again and we’d want an easy way to ignore and ack. Maybe it isn’t a part of the library, but maybe it is a helper data structure and a best practice pattern.

@GLdev

Can we perhaps come up with a couple of real-world examples (say a trivial one - logging? and a more complicated one with a more involved flow) and see how that goes?

That sounds like a great idea!

Since you’ve previously completed a bounty, feel free to get started.

[GLdev]

@skilesare

Theoretically, a boundary node could inspect the message and send back a fake request!

Since this communication will need to change data on the receiving canister, this will have to be an update, thus going through consensus. Let’s see what Dfinity folks say.

[GLdev]

Hi, quick update from me. I’ve framed most of the functionality and the “happy path” is already working:

[Canister rrkah-fqaaa-aaaaa-aaaaq-cai] Sending NEW
[Canister ryjl3-tyaaa-aaaaa-aaaba-cai] Received NEW: 1 - []
[Canister ryjl3-tyaaa-aaaaa-aaaba-cai] Sending ACK
[Canister rrkah-fqaaa-aaaaa-aaaaq-cai] Received ACK: 1 - None
[Canister rrkah-fqaaa-aaaaa-aaaaq-cai] Sending FIN
[Canister ryjl3-tyaaa-aaaaa-aaaba-cai] Received FIN: 1

I’m now looking into state management and timers (there are a bunch of edge cases that need to be caught).

I’ve decided to go with a single update function, and map messages like so:

enum MessageType {
    // A new message that starts a new flow
    NEW {
        msg_id: u128,
        payload: Vec<u8>,
    },
    // Message signaling that a message was received / processed
    ACK {
        msg_id: u128,
        payload: Option<Vec<u8>>,
    },
    // Message signaling that a flow has ended
    FIN {
        msg_id: u128,
    },
}

This allows for extending the capabilities and flows if needed, with more complicated flows, without having to change the did interface. For the moment the candid looks something like this:

type MessageType = variant {
  ACK : record { msg_id : nat; payload : opt vec nat8 };
  FIN : record { msg_id : nat };
  NEW : record { msg_id : nat; payload : vec nat8 };
};
service : {
  asyncFlow : (MessageType) -> ();
}

ACK takes an Optional here, so it can be used both as a traditional ACK (hey, got your message, all is good) and as a response (by adding a payload).

Will keep you posted.

[skilesare]

@GLdev See: Open: ICDevs.org Bounty #39 - Async Flow - One Shot - Motoko - $6,000 - #3 by skilesare

Most of this is relevant for motoko, but it may give some ideas for maintenance functions and composability.

[GLdev]

On it, I’ll engage with @Safik and I’ll share some findings that I’m working on, relating to failure modes shortly.

[GLdev]

Hi, quick update from me:

I’ve integrated the timers functionality. In the tests I was doing I also found a bug in the timers lib when dealing with lots of timers (over 500 in a round) - and that has been fixed . I will re-test with lots of timers when that gets released.

For the moment I’m testing this by dropping some packets (x in 10 packets simply gets dropped at ingestion) and checking that the flow still eventually finishes. All the timeout delays will be configurable (either at init or with every call, TBD) so we’ll figure the best defaults as we go.

I’m now looking into the best approach for calling the appropriate functions when receiving NEW or ACK (treating ACK as a potential response as well) - My goal is to hide as much of the implementation from the user, and make it easy to use in a real-app canister. If a certain pattern is followed on the receiving canister, there could be a way to implement the async_flow to re-use an existing update function, and treat the response of that as the payload for ACK. This would enable a canister owner to re-use code and expose a certain function both as a regular update call, and as an async_flow call.

Will keep you posted.

[Safik]

Hi! GLdev
Did you manage to avoid asynchronous constructs in the class library?

[mparikh]

Not sure about adding a payload to ACK. The issue that i foresee is that does the canister need to ACK the ACK ; because the canister might presumably be processing the payload in ACK and fail? In which case the FIN may never be sent if the semantics of payload in ACK expected the canister to process the payload in ACK.

[GLdev]

Hi, I forgot to add the detailed flow to this topic, I only posted it on the Motoko thread. My take on this is that your case will be dealt by the protocol level, as the canister sending the ACK will re-try after a timeout if it doesn’t receive a FIN message (Item 3 on the Incoming flow below). Let me know if this answers your question, if not I can go into more detail.

Brief flows overview

Outgoing

1. Encapsulate a new message, store payload, etc. Return a msg_id
   (can fail due to canister memory, etc. Ignored for the purposes of this lib)
2. Send Notify(NEW{msg_id, payload})
   (Failure1: the notify call can fail if the canister’s queue is full)
   [Need to re-attempt this call after a set timeout, with a set retry count]
3. Waiting for ACK
   (Failure2: the ACK does not arrive after a set timeout)
   [Need to re-attempt step2, with a set retry count]
4. Received ACK, call processing fn
   (Failure3: the processing fn call traps / panics. Ignored, should be handled by the other canister)
5. Send FIN
   (Failure4: the notify call can fail if the canister’s queue is full)
   [Need to re-attempt this call after a set timeout, with a set retry count]
6. Wait for a set timeout while maintaining the state
   (we need this step in case the other canister doesn’t receive a FIN, and re-sends the ACK. We could simply drop the state at this stage, and reply with FIN to any unknown msg_id. Implementation decision / unsure)

Incoming

1. Receive a NEW message. Create an entry, store payload, decide to accept, call processing fn.
   (can fail due to canister memory, etc. Ignored for the purposes of this lib)
   (Failure1: the processing fn call traps / panics. Ignored, should be handled by the other canister)
2. Processed (Result). Send ACK
   (Failure2: the notify call can fail if the canister’s queue is full)
   [Need to re-attempt this call after a set timeout, with a set retry count]
3. Waiting for FIN
   (Failure3: the FIN does not arrive after a set timeout)
   [Need to re-attempt step2, with a set retry count]
4. Received FIN. Mark task as complete.

[GLdev]

Hmm, I’m not sure I follow. If you’re referring to the function that I call when a NEW or ACK message is received, then for the moment I use sync. If the user of the lib needs to perform async stuff (i.e. inter-canister calls) they can use the ic_cdk::spawn and that should allow them to wrap the async stuff. Were you referring to something else?

[GLdev]

Hi all,

Unfortunately due to unforeseen circumstances I cannot finish this project. I apologize for not updating this sooner.

[Safik]

Hi!
We lost you with @skilesare. Very sorry.
How much progress have you made in the project (as a percentage)? Can you open part of the source code in Rust?

[GLdev]

I’d say ~80% done. I’ll try to figure out a way to access my work computer remotely, and if successful I’ll upload it by the end of this week. If not, Austin will re-assign. Will try to keep in touch anyway.

[Safik]

Ok. It would be good to compare the two implementations. Can you tell me what you have in these 20% percentages? I would like to compare the approaches to implementation. My source code link.
Good luck restoring access to resources. And come back.

[Vitaliy.Klepka]

Hello 2 all.
@skilesare I want to claim this “async-flow-one-shot-rust” if its active. What time limitations do I have for implementation ?

[skilesare]

I will submit you for the bounty and you can get started. Safik has baked the Motoko version so I’m hoping it is a straight forward implementation and that some discussion will come of it.

[Safik]

Good. It is better, of course, to realize your vision. Maybe it will be more optimal. By the way, I will also rewrite my version to Rust

[skilesare]

ICDevs Governance Tool