ICRC-2 Batch operations library

Developers
1

Often times, you might want to make multiple transaction calls to a ledger canister. Doing this sequentially takes a long time.

So, on behalf of a new stealth project :grin:, and while we wait for ICRC-4, I built icrc2-batch, a Motoko ICRC-2 library specifically tailored for parallel execution of allowance, approve, and transfer_from operations.

Using it is as easy as

import ICRC2Batch "mo:icrc2-batch";
import ICRC2Interface "mo:icrc2-batch/IRCR2Interface"

// example (inside an API)...
let icpLedgerCanisterId = "ryjl3-tyaaa-aaaaa-aaaba-cai"; 

let icrc2Actor = ICRC2Batch.ICRC2BatchActor({
  icrc2LedgerCanisterId = icpLedgerCanisterId;
  batchSize = 100;
});

let allowanceArgs : [ICRC2Interface.AllowanceArgs] = [
  { 
    account = { owner = <ownerPrincipal>; subaccount = null };
    spender = { owner = <spenderPrincipal>; subaccount = null };
  },
  ...
];
let allowanceResponses = await* icrc2Actor.icrc2_allowance_batch(allowanceArgs);

(Similarly comes icrc2_approve_batch and icrc2_transfer_from_batch APIs)


The library supports up to 100 parallel async icrc2 operations at a time. Why 100? Itā€™s a ā€œconservativeā€ approach to make sure developers donā€™t overflow their canister output queues.

5 Likes
2

This was my first time really using PicJS and getting it set up in CI. Ran into a few issues along the way, but man is it so much faster than dfx. Thanks so much @NathanosDev!

Also, Iā€™d like to give a shout out to @quint and his testing.mo library.

It uses the interpreter and essentially allows you to run async* tests as unit tests, as long as you provide mock arguments for the async*.

This test is an example of observing a mocked async* function and testing that the arguments passed to it each time are correct.

2 Likes
3

Just published a minor patch update in v1.0.1 of icrc2-batch with some performance improvements based on learnings from @skilesareā€™s questions in this thread.

1 Like
4

IMO icrc2 should be deleted and never mentioned again :smiley: my extreme opinion. Itā€™s a thing that worked for EVM, but it doesnā€™t mean it will work on the IC. ICRC3 and 4 are all you need. Perhaps 3 can be improved a bit by allowing the requester to filter it by from & to Principal.
In the app you could just let people put tokens in a wallet thatā€™s inside the canister, then charge them with sync atomicity and let a tx queue do the ledger synchronization. Security & legally itā€™s the same thing, having access to the tokens with ā€˜approveā€™ or them being in your canister. Approving canisters is actually worse because now wallets need to be able to cancel subscriptions/approvals, and ledgers need to handle ā€˜approveā€™ calls and store more things in memory. Hopefully, the use of ICRC2 will disappear, it is leading to a bad design.
When someone uses a dapp, itā€™s far better to just send a few tokens to it and know thatā€™s as much as you will spend on that dapp, than approve tokens 50 times losing track of what youā€™ve approved while needing more UI to cancel these approvals.
With ICRC3 (even without filtering) and 4 you can receive and send thousands of transactions in one call, it makes everything else obsolete.

2 Likes
5

I donā€™t disagree that ICRC-4 will be a big improvement over ICRC-1/2, but in my experience these things take quite awhile to get rolled out. Do you or others have any idea how far ICRC-4 is from being rolled out?

This might be a discussion for another thread, but there seems to be quite a bit of divergence of opinion with respect to how users want to/should store tokens. Some are pushing for a singular wallet, while others prefer to have a distinct account per app.

Iā€™m not a token expert, nor do I regularly attend the ledger and token standards WG, so this is my naive outsider perspective. Please take it with a grain of salt :sweat_smile:

With most safe wallets, applications do not have permission to spend user funds without explicit permission from the user. This means that until ICRC-2, applications could not implement asynchronous payment workflows unless they actually custodied and controlled user funds.

With the account per app (Internet Identity approach), by default most apps use the delegate principal or account identifier (legacy) for storing tokens on a ledger. The user is required to silo funds by app account, and the app canā€™t access those funds until the user logs in. Again, no opportunity for async payment workflows unless the app directly controlled user funds.

In both the wallet case and account per app (II case), ICRC-2 allows dapps to implement these async payment workflows - regardless of how the user chooses to store those funds. The app does not custody funds on the userā€™s behalf, it is only allowed to move the funds that it has been approved to move, instead of the funds that the user is ā€œwilling to riskā€.

Can you elaborate on this part? How is this the same thing, especially if you are not custodying the funds. Iā€™m thinking of an ICRC-2 approval more like the ability to charge a card monthly on your bill, vs. ICRC-1 (custodied by canister) sending a deposit to your landlord to cover incidental damage. At least to me, these feel very different from a security and legal perspective in terms of liability.

I actually believe ICRC-2 can be greatly improved by a batch method, in the same way that ICRC-1 is being upgraded to ICRC-4. Iā€™m also curious to hear your thoughts around an All or nothing batch transaction icrc standard.

As for the approval storage in memory, this is indeed a concern as well as the ability to view all current approvals. In terms of memory though, there are plenty of vulnerabilities that could arise in programmatically creating a number of new accounts and then dusting those accounts with a trace amount of a token.

Adding in approval traceability per account, upping the cost per approval, and a reasonable outstanding approvals limit per account might help with this in the interim in moving the space from n^2 back to n (with constant multiple), but Iā€™m not convinced that the idea of approvals is flawed. You can either have approvals, or you can end up with people creating the same number of subaccounts for every single app they have a user account in. Itā€™s the same number of accounts and data.

Thereā€™s a time for synchronous payments, and a time for async, but imagine if all of the recurring things you pay for today (AWS bill, Netflix, Internet, Utilities, etc.) required you to move 2-3 months worth of funds in there ahead of time and manually add to those funds every month. Itā€™s extremely inefficient and burdens the user with additional cognitive load. It sort of reminds me thinking about how before the first credit card, individual businesses kept a tab or ā€œledgerā€ of everyoneā€™s account and you had to go in and pay that at the end of each month.

1 Like
6

Sounds like a burn the ships on the shore type of approach that is not realistic in the context of decentralized coordination. Many ledgers including the ā€œprotocol nativeā€ (ICP, ckEtc) ones support the ICRC 1/2 standards today, and itā€™s an improvement over the fractured and unfunctional state of affairs that we were in before those standards.

Still thought provoking for people who are considering ICRC 1/2 integrations. When will 3/4 be supported?

1 Like
7

In the case of ICRC2 we have the dapp frontend (most of the times) getting access to the funds and can spend them while the user is using the dapp. In the other they are in a smart contract (backend). In both cases a malicious actor could take the funds. Smart contracts biggest advantage is that they can be in custody of the funds, while their developers are not - if they are DAO governed or immutable. So when done right, in both cases devs arenā€™t in custody of the funds.

I was thinking of Moonshift. Why is a subscription needed there. A manager decides they want to spend a certain amount of tokens on shifts. They send them to the contract and pay with that. Besides ICRC2 isnā€™t really good for subscriptions, itā€™s just an alternative payment flow.
Wallets arenā€™t even showing the allowances anywhere so we can cancel them. What about II apps, they will need to add interfaces for canceling allowances. And how do we do subscriptions with it at all, allow a dapp to take funds for 2 years ahead and hope it will take it in small parts slowly each month?

Let me show you some of the benefits of a canister holding the tokens compared to a dapp using ICRC-2. You can use devefi-icrc-ledger to do that right now, it uses ICRC-1 and get_transactions. It will work exactly the same way with ICRC-3 and 4 except it will have more throughput.

  • When your canister receives tokens it is now the only thing that can move these tokens, therefore it becomes the new ledger for the tokens it holds. Usually people think that the main ledger is the only ledger and their service canister is lagging behind it with async calls, while it is the main ledger that is lagging behind when it comes to tokens held by a canister.
    The belief that your canister lags behind the ledger leads to unnecessary waiting for transfer calls to come back so your contract can resume its work, making the logic async and prone to errors and attacks. Instead it can move tokens in its memory and queue the transfers.
    Thatā€™s not the case if your canister has allowance with ICRC-2. In that case the devā€™s canister is lagging behind and doesnā€™t become a ledger.

Then you can do this. 4 sync function calls. In this case user is {owner:canister; subaccount: user_principal}. Notice, there is no ā€˜depositā€™ function, itā€™s automatic. The only address the user gets in the dapp frontend is the text format of that account, not a principal.

public func swap(amount) {
  if (ICP.balance(user) < amount or NTN.balance(pool) < amount*rate) return #err;
  ICP.send({from:user; to:pool; amount);
  NTN.send({from:pool; to:user; amount*rate);
}
  • Lets see how it improves payment flows
public func pay_for_service() {
   if (ICP.balance(user) < fee) return #err;
   ICP.send({from:user; to:fee_collector; amount=fee});
   gig.job := #done
}

Itā€™s not making any calls unless the user has funds (and also makes the calls later), compared to the payment flows devs do now, which are not DOS resistant and someone can just force the contract to make calls to a ledger all day flooding and making it inaccessible. The middleware we made takes care of reading the ledger log and updating the balances which means it makes one call every 3 sec that gets up to 2000 transactions. If there are more it can fetch 80,000 for ~13 sec.

With ICRC-2 your payment function will have to await a call to the ledger, because the funds arenā€™t held by it, making it at least ~6 seconds slower for xnet call. In case of BOB a lot more. Now if you are clicking to pay shifts, doing payments with our approach will still take ~3sec (tokens will get delivered later, but they will be delivered 100%), doing it with waiting for async transactions for BOB will take a 20sec per click if it even works.

The only thing ICRC-2 gives us is the ability to share access to tokens with multiple dapps. Makes things complicated and slow. You will be sacrificing UX with it.

How Moonshift could work - You login with II. You make one transaction from your wallet to it. Then you click 30 times on shifts for 1min. Thats it, no wallet dialogs, no additional approvals. It will feel blazing fast. (notice: I tested the dapp few weeks ago, not sure if it was improved since then, but I am guessing if using ICRC-2 it canā€™t just mark a shift as done in the contract and put the tx into a queue, since you arenā€™t sure if the wallet has any funds)

More benefits - Since the tokens arent inside the dapp frontend, but a smart contract. You can add two factor authentication that locks/unlocks accounts or if we are doing large payments, requires 2FA it for every transaction.

1 Like
8

I agree with you technically and in a perfect world where all technology works all the time with no bugs, but the experience where users hits your dApp obliterates it in reality. Some thing like 99% of customers service and negative so social sentiment for adapts jive worked in comes from tokens leaving a persons wallet and the x happens and they donā€™t get what they expected and suddenly youā€™ve rugged them and it is on twitter and someone gets a call at 3am to fight a fire that could be any one of a million things(their Wi-Fi drops, they run out of API calls, youā€™re on the Bob Subnet and your vector only gets to run once an hour, And on and on and on).

By only pulling the funds at exactly the point when the service can provide what is promised, you create atomicity that eliminates a million unknown unknowns.

Iā€™ll likely never build another service that doesnā€™t use the approve - transfer from workflow for actually taking customers funds. I like sleep too much.

1 Like
9

Security & legally itā€™s the same thing, having access to the tokens with ā€˜approveā€™ or them being in your canister.

You certainly should probably not ever approve a canister with ICRC2 that isnā€™t open sourced And/or Blackhoked/dao governed/Someone you really, really trust.

That is why we gave subscription utility to the NNS.

1 Like
10

I donā€™t see it exactly like that. If the approved transfer call fails to return a response but takes the tokens, your canister wonā€™t deliver anything and it has to retry until it gets a deduplication error. Maybe the fee changes at that moment, and then it has to handle that too. Maybe there are 1000s of users smashing the button, it can only do 50 calls a second to the ledger.

If we are arguing about whether having an in-app wallet or an external wallet is best, thatā€™s another thing. With an in-app (smart contract) wallet that has an easy-to-use transfer button and balance, I donā€™t see why users will call at 3 am when they can take their funds out. If your canister didnā€™t deliver, it wonā€™t be taking the tokens and they will be available for transferring. Pretty much every advanced trader is using local Sonic balances and II with ICPSwap because itā€™s way faster. Since ICPSwap has one pool per pair, it has an interface where users can deposit funds in it and then trade locally with ~3-sec update calls and no ledger transfers.
image

But yes, the middleware has to be perfect. The good thing is, the important part is only like 100 lines.

11

Sorry for hijacking the thread with generic ICRC2 talk, let me get back to the point.
I am assuming this library is used by Cycleops. If you had a smart contract account like {owner: canister, subaccount: Principal} then you could easily send out hundreds of ICP transactions to Cycles to canisters automatically. DAOs could send tokens to that wallet address to refuel their canisters. Other smart contracts can send tokens to it too if your canister doesnā€™t rely on additional calls to get notified, but finds out it received funds from the ledger backlog.
I am pretty sure nobody wants to connect every subscription to their only main wallet, but split and budget accounts. In fact, what DAOs I guess would prefer is forwarding the maturity of a neuron toward the account that refuels canisters with cycles. Or BOB miners that get swapped to ICP and then sent to Cycleops account.

12

This is a great point and one I hadnā€™t particularly considered before. Note to self.

With an in-app (smart contract) wallet that has an easy-to-use transfer button and balance, I donā€™t see why users will call at 3 am when they can take their funds out.

This was a huge issue as well. Everyone wanted a browser wallet and didnā€™t want to have to deposit. It was a serious issue and we lost 90+ % of users if they had to move money in. II was a non-starter for non-IC-crypto people.

But yes, the middleware has to be perfect. The good thing is, the important part is only like 100 lines.

Getting lots of new data that makes this even harder on busy subnetsā€¦Iā€™ll try to summarize later.