Does an all or nothing batch transaction standard already exist?
For example, right now, most DeFi applications that make transactions want to assess some sort of fee, which requires 3 transactions
ICRC-2 transfer_from app collects amount + app fee from payer and sends to a holding account
(2 and 3, in parallel)
App sends the amount to the payee destination (could be another user of the app)
App sends the fee to the fees collection destination (app revenue account).
The issue with this is that instead of hitting the ledger directly from the frontend, all these calls need to be made to the canister. So instead of 2-4 seconds for a ledger update on a fiduciary subnet, you get:
2 sec (app canister update call) +
6 sec (cross subnet call plus step 1)
6 sec (cross subnet call plus 2 & 3 in parallel)
= 14 seconds, minimum.
If all or nothing batch transactions were enabled, then we could define the order of transactions and send all together, in order transaction 1, 2, & 3 and if #3 fails then the rest are rolled back since these would all occur synchronously on the ledger,
This would greatly simplify the need to create complex SAGA conditional payout workflows with asynchronous rollback conditions.
Iâm not saying that this canât be built without all or nothing batch transactions, but moreso I think it would eliminate a lot of DeFi complexity, speed up compound transactions, and help out any ICP apps that wants to incorporate just-in-time fees into their business.
ICRC7 already has the concept of (atomic) batch transfers. ICRC4 tries to do the same for ICRC1, but it seems like it is not finalized yet.
All or nothing batch transfers are not always easy to guarantee, in the ICRC7 the token has to indicate explicitly whether it supports this through icrc7_atomic_batch_transfers : () -> (opt bool) query;. I do not see a reason why this can not be done for fungible token too.
Hasnât this type of flow been one of the main issues that was holding back ICP DeFI? Looks like you described atomic vs async DeFi? Im glad to hear itâs being worked on in ICRC-7 and ICRC-4.
You can do atomic if the items are on the same canister. If they arenât on the same canister, then atomic can be done, but via saga and it involves latency. I donât think there is a great solution. It is the same dilemma that every L2 has. They have atomicity once the assets are on that chain, but you have to get them there first.
Maybe. There are some peculiarities in how motoko works with the replica where async calls on the same subnet are âlike syncâ calls most of the time, but I donât think you can guarantee it. (This is how the canpack stuff works with rust libraries).
In theory I think that other pending async class initiated during the round by other canisters on the subnet could be intersperced between your calls, but I think the biggest âdangerâ is when the round is closing and it starts bumping calls to the next round. Timers could be scheduled before your await runs and that could cause atomicity issues.
How do sharded blockchains like MultiverseX achieve all-or-nothing transactions across shards, despite the inherent asynchronous nature of the internet? Are there specific techniques or architectural patterns they employ that could be adapted to improve composability and atomicity on the Internet Computer, considering its subnet architecture shares some similarities with sharding?
So my guess is we are stuck with processing transactions one at a time, which can take up to one minute. This will make it extremely hard to attract DeFi projects which value great user experience. Believe it or not, crypto people see speed as good user experience.
As an ICP user, making an ICP transaction from an app to a ledger on ICP doesnât hold up other ledger transactions from happening. Thereâs general message, ingress, and instruction limits, but ledgers on ICP are quick and can process thousands of transactions per round of consensus (~2-4 sec). The call is quick if the call is made from a principal/delegate identity on the frontend (log in with II, Plug, NFID, any ecosystem wallet).
Canisters can send hundreds of calls (transactions) at a time in parallel to any other canister right now. Thatâs what this icrc2-batch library does. The current ~500 call limit canister A â canister B comes from canister output queue limits, which could be raised in the future.
Referring to point #1, making calls with a delegated identity will always be quick. The slowdown in the case Iâve described occurs when an application wants to perform the transaction asynchronously via an inter-canister call to perform a transaction without triggering a wallet pop-up for UX purposes. For example, the ICRC-2 standard (approve/transfer_from) allows a user to approve another principal (user or app) to spend X amount of funds on their behalf. This enables things like recurring payments, or transferring funds without needing to click yes to transfer funds for every single action, and utilize the X amount of funds that have been previously approved.
In our case, weâd like to be able to trigger these transactions through a canister for improved UX on the userâs behalf (no endless popups), as well as to assess fees on top of each payment that is made. Thatâs the primary reason why weâre going through the canister to perform each of these actions.
With that context in place, having an all-or-nothing batch endpoint, where all calls are made to the same ledger canister would reduce the number of edge cases and complexity that apps need to handle. Projects on ICP want to both provide value and make a profit, and this would help with both!
State changes within a canister are synchronous in nature within a round of consensus, so going from ICRC-4 (batch transactions) which is already in place, to all-or-nothing batch transactions isnât a technical problem. Itâs just something that the ICRC standards community needs to align on and prioritize.
The main point of this forum post from my side was to trigger a conversation, and learn if any work is being done on this front.
Moving forwards is both as easy (and as hard) as getting community consensus to both implement (code) and upgrade (governance) existing token ledgers (ICP, ckBTC, SNS, etc.) to support all or nothing batch transactions. Iâd expect ICRC-4 to come first (batch transactions, without all-or-nothing), but this is a nice addition.
" icrc7:atomic_batch_transfers of type bool (optional): true if and only if batch transfers of the ledger are executed atomically, i.e., either all transfers execute or none, false otherwise. Defaults to false if the attribute is not defined."
The atomic batch transfer referenced in this quote is stored this in the canisterâs metadata, which makes me thing that all batch transactions on the canister are either atomic or not. Iâm not sure why there isnât the option to perform both atomic and non-atomic batch transfers on the same canister/ledger , but maybe thatâs a point for discussion moving forwards.
If an ICRC-7 ledger supports atomic transaction, thereâs no need for non atomic calls since both would basically result in the same behavior and response in that ledger implementation.
With only one key difference, an atomic ledger will immediately return an error element. But even a non atomic ledger can throw a single error as response which will need to be handled just the same.
So basically an atomic ledger doesnât actually behave differently from the perspective from a client, it returns a list of 0 or more response with possibly an error. But if you know the ledger is atomic (by checking this metadata field) you additionally know that it wonât return partial lists of responses.
Does this assume the same subnet? I canât ever get more than like 12 async calls at a time before hitting the instruction limit on most of my tests. Curious if Iâm doing something wrong.
It is a small point, but we should be clear that these batches are being spanned across multiple rounds and you canât assume elsewhere in your code that other stuff isnât happening while you are waiting multiple rounds for all the items to get filed.
The bottleneck I would have expected you to hit in this example around 200-500 is due to cycles reserved for outgoing calls, since canisters on the Motoko playground have a limited cycles balance, but that doesnât seem to be occurring
I would have then expected you to reach the canister output queue limit at around ~500 outgoing calls, but I just ran this code with 600 calls, which originally made me think there is some optimization for calls directed back at the same canister.
However, then I set up this 2 canister example in the playground just to make sure.
Seems to work fine with 600 outgoing calls, which I did not expect at all . Iâd still stick to less than 500 outgoing calls at a time to be safe and implement batching logic, but hopefully these examples are helpful.
@dsarlis or @claudio - any idea why this example allows me to queue up more than 500 calls at a time between canisters?
In your example you have the actual await inside of a helper which means that when it actually awaits each future it is doing an individual await and resetting your instruction limit each time. It seems creating an async* future does not do the same calculation that a straight-up async future produces. Interesting.
It makes sense that you are not running out of cycles because you are doing them all inline even though it seems like you are doing it in parallel. (I understand that functionally it may not matter). If you look at my updated example: https://m7sm4-2iaaa-aaaab-qabra-cai.raw.ic0.app/?tag=2787402379 you will see that after running test() and then getSum that only about 26 executions happen per block.
This may be fine if you are calling something on the same canister, but I would imagine if you were calling something on another subnet, because each await is halting execution until it resolves, that it would take 500*6 seconds to get through them all. (sure enoughâŚif you check out Motoko Playground - DFINITY youâll see that each balance check takes 6 seconds and they are not going in parallelâŚclick getSum while waiting for test to finishâŚIâm guessing the response will time out eventuallyâŚbut I wonder what happens on the canister? I guess it keeps going). (edit: I got Server returned an error:
Code: 400 ()
Body: Specified ingress_expiry not within expected range: Minimum allowed expiry: 2024-09-05 16:46:07.283220654 UTC, Maximum allowed expiry: 2024-09-05 16:51:37.283220654 UTC, Provided expiry: 2024-09-05 16:46:00 UTC) âŚbut that may be just because it was an ingress call.
There does seem to be some nice accounting going on if youâre calling the same subnet/canister here though so it is a nice pattern to use to keep from having to process the futures in batches like I had been doingâŚbut as soon as you go across a subnet boundary youâll want to process in batches. See the speed improvement on Motoko Playground - DFINITY where I actually do get 8 balance response about every 6 seconds.
If you go above 25 calls in parallel, youâll receive this error
Call was rejected:
Request ID: 35071e1b663a96b69ca970787b13653e09cbbec4b40f747c3aaf741e021fea00
Reject code: 4
Reject text: could not perform remote call
This is actually due to the canister running out of cycles that are reserved when a call is made (20B per call). Since the Motoko playground limits how many cycles are on a canister at any point in time, running this with a canister on mainnet that has more cycles on it ensures that more calls in parallel can be made.
Then you should run into the next xnet call limit at 500, which is the canister output queue limit. You can get around this by batching calls.
, but maybe thatâs a point for discussion moving forwards.