Big Map (for scaling canisters across subnets)

Summary

The DFINITY Foundation is building out an advanced scaling solution for linking an unbounded quantity of Canisters together. However, the Foundation is first prioritizing increasing Canister smart contract storage which will address immediate developer pain points.

Status
Discussing

What you can do to help

  • Ask questions
  • Propose ideas
5 Likes

Can you explain how this will work?

I assume it’s quite different from the 300 GB stable memory proposal, but I’m not clear on the details.

1 Like

Yes, this is different because the 300G solution can only scale upto 1 subnet.

This proposal is about having a canister which can be so large that its state is be preserved across many subnets.

I am not close enough to the implementation details of how this would work, but it would essentially mean that a canister could infinitely* auto-scale as it needs to.

Once we put some researchers and engineers that can answer this better, I will link them to this thread so they can answer questions and be better prepared to go into low level detail.

2 Likes

Yeah, more details would be very helpful! This does sound like a “game changer” but at this point it’s so vague I will postpone the celebration party until those details are made available.

1 Like

Would be great if anyone working on this could share any blockers they’ve run into!

1 Like

Update:

This project will come after Increased Canister Storage: Increased Canister Storage - #13 by flyq (and they share some of the same people), so expect this project to come after once we see the impact of what increasing the storage does (to gauge how urgent it is to scale beyond a subnet).

You can see the updated summary and thread on Increased Canister Storage: Increased Canister Storage - #13 by flyq

2 Likes

Hopefully this also comes after increasing the heap on canisters. Increasing the stable memory is only the first step to a seamless storage experience, honestly just increasing the stable memory is still going to be difficult to deal with, not ideal, not the end.

2 Likes

Agreed. Our current thinking is to make sure that the ramp-up to 300 GB is deployed and flawless before jumping too deep into on this larger, much more infinite scaling…

1 Like

Some remarks based on the above discussions.

Storage capacity of a single subnet

A remark about the 300GiB limit. This is just the current capacity of a subnet. There is no reason why we cannot have subnets with larger capacities. In fact, it is indeed possible for one to make improvements to the protocol so that the IC has to keep fewer copies of the state around which will allow a subnet to support a larger capacity. And also in future, if a subnet has beefier nodes with larger disks, that will also allow it to support more capacity.

Other issues with scaling to a single subnet

One thing to keep in mind is that even if one were to create a canister with a 300GiB state, they might hit other types of scalability limits. A canister can currently only have a single thread of execution (because wasm is currently single threaded and multi-threaded deterministic execution would a challenging problem to solve). So if a subnet hosts a single canister with 300GiB of storage, it cannot host any other canisters. A subnet can execute multiple canisters concurrently but this subnet will only ever be able to execute a single canister at a time.

In other words, beyond scaling a canister’s storage capacity, one may also want to scale a canister’s update throughput capacity. And currently, the only way to do that is to split the canister into multiple canisters. So depending on your workload, even if all your data could fit in a single canister, you may still need to split it into multiple canisters.

Larger wasm heaps

@lastmjs: I would love to hear more about your use case where you need a wasm heap larger than 4GiB when you have access to a large stable memory. In my mind, with a large stable memory, canisters would typically use stable memory as the main storage and the wasm heap as a scratchpad. When executing a message, they will copy over the relevant data from stable memory to heap, make changes to it, then copy the updated data back to stable memory. So in this model you only need a larger heap if you plan on copying over more than 4GiB data from stable memory which given the current message execution limits, you cannot do anyway.

Thoughts on BigMap

The DFINITY org had done a PoC demo before launch. It was done by having an indexing canister that tracks which keys are hosted on which data bucket. So the user would send a request to the indexing canister for the key, which would then forward it to the right data bucket, which would then send a response.

This implementation could benefit from a fully functioning inter-canister queries implementation but there are ways to get around it.

In short, if someone in the community wanted to take a whack at building their own BigMap implementation, that should already be possible today.

One such whack at this is here!— Mini BigMap library

1 Like

My use cases are mostly based around the developer experience of library and app developers alike. Right now, if you can limit yourself to 4gb, orthogonal persistence provides an elegant API for storing arbitrary data in arbitrary data structures. You can pull standard data structures out and simply start using them. I do this in Sudograph, composing standard BTreeMaps to create a capable (yet simple) relational database. I can use the standard APIs to read and write from these maps. I have one logical block of memory that I don’t need to think about in IC-specific ways.

The problem with having a small heap and copying data from stable memory is exactly that extra step. Just as an example, with a small heap, searching through a BTreeMap I wouldn’t be able to use the BTreeMap data structure as something I have full access to in local memory. I would have to write some kind of glue code to copy over pieces of data one at a time to search through.

That sounds simple enough, but repeat this for every possible data structure and every possible operation. The most elegant solution I can think of for developer experience (I am including the developers of libraries that might abstract these things away) is to have one large block of memory that is accessed through the standard APIs of the language the developer is working in.

If we can’t do this, I imagine we might have to come up with specific abstractions for every new type of data structure and operation, causing the community to create and maintain a whole collection of IC-specific data structures or libraries, where otherwise we would not need so many.