Some remarks based on the above discussions.
Storage capacity of a single subnet
A remark about the 300GiB limit. This is just the current capacity of a subnet. There is no reason why we cannot have subnets with larger capacities. In fact, it is indeed possible for one to make improvements to the protocol so that the IC has to keep fewer copies of the state around which will allow a subnet to support a larger capacity. And also in future, if a subnet has beefier nodes with larger disks, that will also allow it to support more capacity.
Other issues with scaling to a single subnet
One thing to keep in mind is that even if one were to create a canister with a 300GiB state, they might hit other types of scalability limits. A canister can currently only have a single thread of execution (because wasm is currently single threaded and multi-threaded deterministic execution would a challenging problem to solve). So if a subnet hosts a single canister with 300GiB of storage, it cannot host any other canisters. A subnet can execute multiple canisters concurrently but this subnet will only ever be able to execute a single canister at a time.
In other words, beyond scaling a canister’s storage capacity, one may also want to scale a canister’s update throughput capacity. And currently, the only way to do that is to split the canister into multiple canisters. So depending on your workload, even if all your data could fit in a single canister, you may still need to split it into multiple canisters.
Larger wasm heaps
@lastmjs: I would love to hear more about your use case where you need a wasm heap larger than 4GiB when you have access to a large stable memory. In my mind, with a large stable memory, canisters would typically use stable memory as the main storage and the wasm heap as a scratchpad. When executing a message, they will copy over the relevant data from stable memory to heap, make changes to it, then copy the updated data back to stable memory. So in this model you only need a larger heap if you plan on copying over more than 4GiB data from stable memory which given the current message execution limits, you cannot do anyway.
Thoughts on BigMap
The DFINITY org had done a PoC demo before launch. It was done by having an indexing canister that tracks which keys are hosted on which data bucket. So the user would send a request to the indexing canister for the key, which would then forward it to the right data bucket, which would then send a response.
This implementation could benefit from a fully functioning inter-canister queries implementation but there are ways to get around it.
In short, if someone in the community wanted to take a whack at building their own BigMap implementation, that should already be possible today.