If we use icrc-3 for all kinds of logs: Ledger logs, dex logs, app logs, user activity logs, etc. It makes sense if we can have multiple logs inside one canister and not try to get everything out through one single log.
account: opt Account could become channel: opt Blob and that can fit an account, text path, and many other things, even have filters, range, etc inside.
I usually have multiple logs in my canisters. The ICRC ledger can just use one, but the spec can allow more.
If we look at something like Linux, it has many logs, and tailing logs separately is essential.
Now imagine all Linux logs were merged inside one and you had to pipe GB of logs through a filter to get what you want.
Also, some logs like the error log wonât need certification or hashing. Isnât certification only needed by off-chain consumers? While hashing is helping observers quickly verify that upgrades didnât tamper with the log? If we have another canister that takes the raw logs and certifies and hashes blocks, wouldnât that offload the main canister and result in a modular architecture that provides the same guarantees?
Something like the asset canister, but instead it handles logs, creates archive canisters, hashes, certifies, etc. The main canister that produces them then doesnât have to do any of these.
Letâs not forget the ICRC-3 log is more than just a log, itâs a single blockchain that represents the first, last and all states in between the ledger has ever been in. The blockchain part guarantees that a block in between cannot be modified without invalidating the whole chain.
I donât think splitting the log into a per user, per sender etc logs makes this less complex, it would actually create multiple sources of truth, which is something you really really want to avoid.
So personally Iâm still leaning towards having a single log that has all canister state mutations ever made. If logs are needed for other purpose besides keeping track of all canister state mutations since inception, I suppose this should be something separate.
As for an index canister that e.g. does allow looking up with filters, I donât think this canister has to be a separate canister from the ledger canister in all cases. Though I would personally opt to make it a separate canister based on the ICRC-3 chain simply because itâs something that can evolve, have more features in the long run, index multiple ledgers in combination etc.
Icrc-3 interface being able to output multiple logs/streams/events/blocks in a chain doesnât mean the ledger has to use more than one. Itâs up to developers. Like @levi mentioned, it can be reused by the indexer. It can also be used by log aggregators.
The question is, are we going to use it as a log retrieval mechanism or it will be just for ledgers?
Because apps do need multiple output logs/streams as a major form of inter-canister communication.
Take this example - your app adds entries, but they are app-specific and not standard. They can be used to replay state, but other parties canât understand these, or are not interested in your custom logs enough, or just donât want to use something that devs can change at their whim and break integrations.
So the app also adds the same events in standardized schemas, which are not used for replaying state, but for communication. Maybe it even pushes out the same information in 3-4 different schemas. Why merge them all in one if icrc-3 is used for these?
Sometimes itâs better to follow a stream with changes: Your app needs to follow 50k ledger accounts. Making 50,000 icrc1_balance_of calls every 2 sec will be impossible. If you follow the log you will get this done with 1 call every 2 sec.
if the previous block : Value of Map variant contains phash field, do we skip this field during the hashing for the current blockâs phash? or do we hash everything from the previous block as the phash of the current block?
I donât think itâs specifically mentioned in the ICRC3 standard that ALL transactions (mint/ txfr etc) which change a balance MUST be recorded. I found an issue with a couple of community canisters where they have a pre-mint which is not recorded in the ledger history.
In practical terms the first transaction of a ledger canister history MUST be a mint transaction? Some canisters however are starting with a transfer which doesnât really make sense and isnât auditable in terms of what balance the account doing the transfer has access to.
hi. do i have to create a new hash_tree (or make it empty) every time I create a new block?
like this
// ... creating new block for id = i
hash_tree := HashTree.empty(); // <-- do i need to do this?
hash_tree := HashTree.put(
hash_tree,
[encodeUtf8("last_block_index")], // path from root
encodeLeb128(current_block_id)
);
hash_tree := HashTree.put(
hash_tree,
[encodeUtf8("last_block_hash")], // path from root
hashValue(current_block)
);
It depends on what else you might have in your hash tree. ICRC3 enables you to have both an ICRC3-certified hash for your log as well as other certified data. To date weâve been passing in @nomeata 's cert tree as an environment parameter and adding these items to the tree. Once youâve set them you can setCertifiedData.
In our Fungible Tokens, when we set up icrc3 we pass this token level tree to the icrc3 module:
Iâm new to the discussion but see a great opportunity here.
In many ways, a canister can be thought of as a blockchain in a traditional sense, and in the web3 ecosystem, there has been built a lot of infrastructure around consuming and indexing blockchains by transaction or event log data. Some examples would be Dune, Nansen, Dappradar, Tokenterminal, Dexscreener, Dextools, etc. (There are many more).
Dapps on ICP are missing out quite a bit on discoverability and credibility because of this.
Iâm wondering if icrc3 + tooling like Motoko Rechain (Blockchain Middleware - ICRC-3 related) could be a great enabler to create a simple and standard mechanism for dapps to expose an event log that off-chain infrastructure can consume. From my perspective, itâs also not necessary for each chain to keep the entire log on-chain forever.
For these off-chain integrations, I think, it would also be useful to standardize a http_request method to fetch the chain.
We believe Rechain would be a highly useful middleware, aiding developers in logging record types and simplifying development processes. It will also facilitate users in querying their records.
We consider several features to be required:
Multi-dimensional filtering. For example, taking our transaction records as an example, there are key fields such as the userâs principal, transaction token0, transaction token1, type, etc. that need to be filtered.
The availability of a unified dashboard for querying. where users can access their relevant transaction (transfer) data from the same dashboard.
Performance in storing records. For instance, in swap transactions, many logs are generated. However, considering the memory capacity of canisters, storing these logs would require a separate log canister, However, cross-canister calls are slower, resulting in a poorer user experience. Therefore, for many business logs, we only retain the final results and not the process logs.
If Rechain can achieve low latency, approaching the speed of internal canister calls, we would be willing to store business logs in Rechain.
And we encountered a situation that might or not might related to what you mentioned, but we thought it may be an optimization point for ICRC standards. We encountered some ICRC tokens where the volume of transactions was too high, causing the Archive Canister of the token to exhaust its default cycles, while the Ledger Canister of the token still had cycles. This resulted in uninterrupted token swaps, but transfer records were not recorded, making reconciliation very complex for us. However, for non-SNS tokens, we currently have no way of finding these Archive Canisters of the tokens, so we cannot get warnings that cycles are about to be exhausted.
Thanks for letting us know what you need.
I think Rechain should be kept as simple as possible and just handle reducing state(optional), (ICRC3) transactions, hashing, archives and provide a framework for plugins.
(1) Anyone can make a module that connects to it and indexes transactions - inside the same canister or another âindexâ canister. The delay will be max 2 sec if any - when on the same subnet. RxMoDb can easily index these fields. If placed inside the same canister - maybe wonât be a good idea before Motoko gets bigger stable memory.
Something like this will work:
mops add a_custom_module
import the module, configure it and add it to the array of reducers (plugins)
all dispatched transactions will go through it
Rechain in one canister will be able to follow rechain in another, so devs decide where they want to install the reducer plugins.
I am also guessing, nobody will want to rewrite their whole state management and put everything inside reducers, so that is optional. You can just dispatch transactions without any reducers and they will be sent straight to the ICRC-3 chain.
(2) I think multiple dashboards will be made by the community once ICRC3 gets out
(3) The way it works now canisters write blocks inside local memory, so no cross-canister calls and it shouldnât result in poor UX. Indexers are periodically fetching blocks from the main canister and only query archives if they are too far behind.
why is the callback have to return { log_length; blocks; archived_blocks } : GetBlocksResult, instead of just [{ id : Nat; block : Value }]? does this mean all the remote/archive canisters have to keep track of the amount of blocks from base to tip for log_length and also keeps track of other remote/archive canisters for archived_blocks?
how about we make all these query endpoints to be of composite query so the archive canisters can query the main ledger for total block size, and the main ledger can query the archive canisters for their start, end? this way, archived_blocks wont be needed anymore since the main ledger can query archive canisters and include the result in blocks.
edit: ok for (2) i know why we cant make it composite query because composite queries can only be triggered by browser agent or dfx only, while these interfaces are for all including intercanister calls
Thanks for the feedback. Looks like the consensus is that itâs better to leave icrc-3 with just the block log and possibly put filters in a different standard. Also there can be implementations of icrc-3 for block logs without accounts, since icrc-3 is generic like that.
Yes for the response returned by the ledger canister. I donât know about the response returned by the archive canisters.
The callback returns the archived_blocks field so that the archives can have their own archives (nested archives).
Hmm good question. It could be that an archive canister should return the length of the blocks within that archive-canister, but Iâm not sure if thatâs how itâs meant to be.
