The ledger canister spawns archive canisters to deal with storage constraints.
The index canister doesn’t seem to do this, and it’s unclear what will happen when it reaches storage limits down the line.
This is relevant for developers seeking to build resilient, future-proof services that they would like to blackhole without worrying about system-level contracts substantially changing years down the line.
Will the existing index canister always be capable of providing the very oldest and the most recent transfer to a specific account (along with all transfers inbetween)? My hope is that it would be upgraded to utilise multiple archive indexers rather than become deprecated in favour of a new index canister interface and cansiter id. I’m wondering what the current thinking is about that within DFINITY.
Good question! We’ve touched on this topic internally a number of times in the past few years as part of discussions on the ledger suite architecture in general. Some key points regarding the ICP ledger suite:
- The ledger only stores up to the latest 2000 blocks, so it uses negligible amounts of memory (around 220MB of stable memory, mainly for account balances and allowances, and 30MB of heap for transactions in the 24h deduplication window).
- The bulk of the blocks are stored in its four archives - the first three archives use around 1GB, whereas the active fourth archive is currently using around 2.4GB.
- The index stores all the blocks in stable memory (around 13.7GB).
- The maximum storage limit for stable memory for a single canister is 500GiB, whereas the storage limit for all canisters on a subnet is currently 2TiB.
- The
tdb26 NNS system subnet is currently using around 27GB of storage.
Given the above, we don’t see an immediate danger of storage running out for the ICP ledger suite - not for the ledgers, archives, nor the index. If/when this does become an issue in the future, some options would be:
- Migrate some archive canisters to other subnets - keeping the active archive on the same subnet as the ledger would probably still be a good idea. An up-to-date index would very rarely, if ever, need to fetch blocks from an archive, so the additional challenge of having to make cross-subnet calls should not be a problem in practice.
- Migrate the index to another subnet. This would alleviate storage pressure, but would otherwise not necessarily be a good move, since the index would have to make cross-subnet calls to contact the ledger.
- Shard the index across subnets, e.g., by storing older blocks in an index archive.
- Do not store blocks in the index, but instead either serve them directly from the ledger or archive, or change the API to only return pointers, and the clients would have to connect to the ledger and/or archives to get the blocks.
- Going in the opposite direction - getting rid of archives entirely, and integrating the index into the ledger could also be an option, as it would reduce the total storage used by the ledger suite. However, then the 500GiB per canister would be a hard limit, and we would lose the flexibility to shard across subnets, or load balance across multiple canisters.
For ICRC ledger suites, most of the above also applies. The DFINITY-managed chain fusion ledgers (e.g., ckBTC) are deployed on the pzp6e fiduciary subnet, which is currently using around 63 GiB of storage, and the SNS ledger are on the x33ed subnet, which is using around 22 GiB of storage, so neither single canister, nor total subnet storage is currently an issue for them. Independently deployed ledger suites also exist on other subnets, and on some, the total subnet storage usage will certainly be larger than on the aforementioned subnets. For such ledger suites, the recent canister migration feature could be an option to move them to a less busy subnet while preserving the canister ID.
All of the above would require some action from a controller (e.g., upgrading a canister, or migrating a canister to another subnet), so blackholing would still be a problem, but that’s not specific to ledger and index canister, and is perhaps best tackled as a separate discussion.