Summary
The DFINITY Foundation is building out an advanced scaling solution for linking an unbounded quantity of Canisters together. However, the Foundation is first prioritizing increasing Canister smart contract storage which will address immediate developer pain points.
Status
Discussing
What you can do to help
Can you explain how this will work?
I assume it’s quite different from the 300 GB stable memory proposal, but I’m not clear on the details.
Yes, this is different because the 300G solution can only scale upto 1 subnet.
This proposal is about having a canister which can be so large that its state is be preserved across many subnets.
I am not close enough to the implementation details of how this would work, but it would essentially mean that a canister could infinitely* auto-scale as it needs to.
Once we put some researchers and engineers that can answer this better, I will link them to this thread so they can answer questions and be better prepared to go into low level detail.
Yeah, more details would be very helpful! This does sound like a “game changer” but at this point it’s so vague I will postpone the celebration party until those details are made available.
Would be great if anyone working on this could share any blockers they’ve run into!
Update:
This project will come after Increased Canister Storage: Increased Canister Storage - #13 by flyq (and they share some of the same people), so expect this project to come after once we see the impact of what increasing the storage does (to gauge how urgent it is to scale beyond a subnet).
You can see the updated summary and thread on Increased Canister Storage: Increased Canister Storage - #13 by flyq
Hopefully this also comes after increasing the heap on canisters. Increasing the stable memory is only the first step to a seamless storage experience, honestly just increasing the stable memory is still going to be difficult to deal with, not ideal, not the end.
Agreed. Our current thinking is to make sure that the ramp-up to 300 GB is deployed and flawless before jumping too deep into on this larger, much more infinite scaling…
Some remarks based on the above discussions.
Storage capacity of a single subnet
A remark about the 300GiB limit. This is just the current capacity of a subnet. There is no reason why we cannot have subnets with larger capacities. In fact, it is indeed possible for one to make improvements to the protocol so that the IC has to keep fewer copies of the state around which will allow a subnet to support a larger capacity. And also in future, if a subnet has beefier nodes with larger disks, that will also allow it to support more capacity.
Other issues with scaling to a single subnet
One thing to keep in mind is that even if one were to create a canister with a 300GiB state, they might hit other types of scalability limits. A canister can currently only have a single thread of execution (because wasm is currently single threaded and multi-threaded deterministic execution would a challenging problem to solve). So if a subnet hosts a single canister with 300GiB of storage, it cannot host any other canisters. A subnet can execute multiple canisters concurrently but this subnet will only ever be able to execute a single canister at a time.
In other words, beyond scaling a canister’s storage capacity, one may also want to scale a canister’s update throughput capacity. And currently, the only way to do that is to split the canister into multiple canisters. So depending on your workload, even if all your data could fit in a single canister, you may still need to split it into multiple canisters.
Larger wasm heaps
@lastmjs: I would love to hear more about your use case where you need a wasm heap larger than 4GiB when you have access to a large stable memory. In my mind, with a large stable memory, canisters would typically use stable memory as the main storage and the wasm heap as a scratchpad. When executing a message, they will copy over the relevant data from stable memory to heap, make changes to it, then copy the updated data back to stable memory. So in this model you only need a larger heap if you plan on copying over more than 4GiB data from stable memory which given the current message execution limits, you cannot do anyway.
Thoughts on BigMap
The DFINITY org had done a PoC demo before launch. It was done by having an indexing canister that tracks which keys are hosted on which data bucket. So the user would send a request to the indexing canister for the key, which would then forward it to the right data bucket, which would then send a response.
This implementation could benefit from a fully functioning inter-canister queries implementation but there are ways to get around it.
In short, if someone in the community wanted to take a whack at building their own BigMap implementation, that should already be possible today.
One such whack at this is here!— Mini BigMap library
My use cases are mostly based around the developer experience of library and app developers alike. Right now, if you can limit yourself to 4gb, orthogonal persistence provides an elegant API for storing arbitrary data in arbitrary data structures. You can pull standard data structures out and simply start using them. I do this in Sudograph, composing standard BTreeMaps to create a capable (yet simple) relational database. I can use the standard APIs to read and write from these maps. I have one logical block of memory that I don’t need to think about in IC-specific ways.
The problem with having a small heap and copying data from stable memory is exactly that extra step. Just as an example, with a small heap, searching through a BTreeMap I wouldn’t be able to use the BTreeMap data structure as something I have full access to in local memory. I would have to write some kind of glue code to copy over pieces of data one at a time to search through.
That sounds simple enough, but repeat this for every possible data structure and every possible operation. The most elegant solution I can think of for developer experience (I am including the developers of libraries that might abstract these things away) is to have one large block of memory that is accessed through the standard APIs of the language the developer is working in.
If we can’t do this, I imagine we might have to come up with specific abstractions for every new type of data structure and operation, causing the community to create and maintain a whole collection of IC-specific data structures or libraries, where otherwise we would not need so many.
I’m not sure I fully understand this. I have a couple questions:
I’m curious why DFINITY hasn’t open sourced that PoC demo, even if it isn’t feature complete. Is it not secure? It’d be very useful to see what that code looked like, I think.
Thanks for the reply @lastmjs . I understand now. I agree this is a big issue with the current setup. We obviously wouldn’t want to ask the developers to reinvent all the data structures with stable memory as the backing storage. I also agree that wasm64 would help address this problem. Do you then have any thoughts on how we could handle the upgrade story for when a canister has a huge wasm heap that it needs to serialise in the pre_upgrade hook and deserialise in the post_upgrade hook?
- How do subnets prevent a single canister from monopolizing the entire 300 GB stable memory?
All non-empty canisters use a bit of storage.
Today, if a single canister is willing to pay for all 300 GiB of storage on the subnet and manages to get in and use all the storage before other canisters can use any storage, then no one else will be able to create new non-empty canisters on that subnet.
I do see this as a problem and would love to hear thoughts on how the situation could be improved.
- Why can subnets not execute multiple canisters in parallel? I thought a single canister was single-threaded and replicated across all nodes in a subnet, but potentially each replica node could run independent wasm modules in separate threads. After all, what’s the purpose of having such beefy nodes if their multiple cores aren’t being used?
- Adding on to the above, if subnets are truly single-threaded in execution, then scaling to multiple canisters wouldn’t help if those canisters were all colocated in the same subnet. Wouldn’t it be the same as running it all in a single big canister, since only one update is processed by the subnet at a time anyways? (If the canisters are distributed across multiple subnets, then this would increase update throughput. However, we as developers can’t choose which subnet we deploy to right now anyways.)
My point was that if I manage to create a single canister on a subnet that is using all 300GiB of storage on it, then no one else can create non-empty canisters on that subnet. So even though the subnet has capacity to execute a bunch of canisters in parallel, there is exactly 1 non-empty canister available on that subnet.
I’m curious why DFINITY hasn’t open sourced that PoC demo, even if it isn’t feature complete. Is it not secure? It’d be very useful to see what that code looked like, I think.
I have filed a request to open source the demo. I have already gotten a bunch of approvals and hope to get the remaining approvals this week.
That would be incredible, thanks.
Also your point about subnet concurrency makes sense.
I want to give props that Akhi for pushing on this. It can be a lot of “extra homework” above and beyond one’s already long daily work to go through the security reviews for open sourcing so it’s no small feat.
Please see Releasing the source of the BigMap PoC demo. The code is now released.