Thanks for sharing @roman-kashitsyn!
Before I go digging through those repos, could you share how those 2 canisters address (or aim to address) the current limitations?
Before I go digging through those repos, could you share how those 2 canisters address (or aim to address) the current limitations?
Internet Identity is pretty straightforward: it divides stable memory into fixed-size chunks (2KiB at the moment) and employs a simple mapping to translate user numbers into chunk indices. When a user data is read/updated, the canister goes directly to stable memory to read/update the corresponding chunk. See storage.rs for more details.
The canister also assumes that it’s assigned a range of user numbers to handle, and the scaling scheme we have in mind is to have multiple such canisters as storage + an index canister that stores the mapping from user number range to canister id. 4GiB of stable memory is enough to support >1M of user numbers for now. We might need to revisit the limits as we start supporting Windows Hello which uses less storage-efficient encryption schemes.
Certified assets is supposed to store assets of a web-app, so it assumes that everything fits into the main memory at the moment. Though it’s not a fundamental limitation: the certification if efficient enough to supports hundreds of thousands of assets, and the asset bodies could be stored in the stable memory. I have a design that turns stable memory into a HashMap<Bytes, Bytes> that requires no work to persist across upgrades, but I haven’t implemented it yet. That would work nicely for something like assets canister.
5 Likes
This thread abstracts well. The issue is that other ‘folks’ don’t document their stuff with Internet Computer disciplined spice in mind. So expectation is updates will break … this is the edge eh?
Stable memory interfaces FTW.
To be filed under the heading the best ideas are often plagiarized… I would note that for new 32 bit designs the external address bus is 36 bits wide giving a larger address space than 4 GB & the external data bus is 64 bits wide primarily in order to permit a more efficient prefetch of instructions and data.
I wish I had more to contribute but haven’t really gotten into the technicals too much myself. However, I’ve recently come across FASTER which seems like it may have a lot of ideal properties for the internet computer.
If, as you say, stable memory limit is much easier to expand, I believe, the team should definitely go that way.
In this case, we could use stable memory as a permanent and (hopefully) infinite storage. But in scenarios when cost efficiency is the case, we could use linear memory as some kind of caching mechanism, which is limited in space, but much faster than stable memory.
I think, this would fit almost any application, because it is very much like web2.0 setup.
2 Likes