Hey everybody,
A few weeks ago, we had the 4th ICP.Lab at the DFINITY HQ in Zurich to discuss pain points and potential solutions with teams that push the boundaries concerning Storage and Scalability on the Internet Computer.
For this ICP.Lab, we’d like to do something new, and provide summaries for (most of) the sessions for anybody who is interested. This is a long post, so get yourself a cup of coffee.
Content
- Introductions and main pain points of the participating teams
- Stable Structures & Memory + Data structures and schema evolution
- Scalability: Current limitations and improvements
- Scalability 2 - Multi-canister architectures
- Immutable Storage API
- Backup and restore
- Internet Computer as CDN
- Queries & Query charging
- Encryption and Data Privacy
- Partnerships between the teams
Introductions and main pain points
To start off the week, each project was asked to present themselves briefly and present their main pain points. In the following summary, we list the pain points mentioned.
Origyn
- Subnet routing/Subnet Distribution - need to roll our own (self-directed subnet assignment)
- Caching - need to roll our own
- Alternative routing - prptl.io
- Stable Storage?
- Cost clarity (storage pricing/storage subnets, queries)
- Easier backup and restore
Anvil/Internet Base
- The agility of (Motoko) data structures. How to enable fast queries on data structures that evolve over time?
CanDB
- Canister cloning (in order to do efficient sharding/partitioning)
- ICQC could enable a more performant CanDB canister client (Motoko/Rust)
- Canister groups & Subnet Splitting help canisters remain on the same subnet
OpenChat
- Support more canisters per subnet
- Threshold Key Derivation to support e2e encryption
- SEV-SNP so user data is more secure when e2e encryption is not enabled
- Fire and forget HTTPS outcalls from canisters
- Inter subnet query calls (+ capability-based security model)
- Upgrading canisters doesn’t require stop/upgrade/start but instead queues ingress messages during upgrade
- Cost to provision and upgrade canisters
- Canister hook on low cycles balance
Hot or Not
- AuthZ mechanism for canisters belonging to the same project (Forum post)
- Inter subnet query calls
- Cost of canister creation for user-per-canister dapps
- Native backup/restore mechanism (Forum Post 1 and 2)
- Handle chunking at the protocol level instead of the application level
- Better primitives of working with stable memory
- Safer primitives for canister upgrades using pre/post upgrade hooks
- Specialized subnets
- Low-replication storage subnets that aren’t globally replicated
- GPU support for hardware accelerated
- Video encoding/decoding
- Training neural networks
Portal
- The dream is boundary nodes functioning as CDNs
- Caching on the boundary nodes
Signals
- Mobile iOS issues with streaming videos
- Usability of stable memory
- Complexity in migrating large amounts of data over the network (Cloning/Partitioning)
- Best practices for testing
Stable Structures & Memory + Data structures and schema evolution
Storing persistent data in the Wasm heap has two main drawbacks: upgrades become riskier because the entire contents needs to be serialized to stable memory (without bugs!) and storage to 4GB. We recommend using stable memory for persistent data because it allows for safe upgrades and 48 GB of storage (soon to be increased!).
Wasm-native stable memory is improving the performance of stable memory. It’s still over 10x slower in some cases, but other cases are only ~2x worse and there’s still a bunch of work to be done that can bring it close to the heap.
The ic-stable-structures and ic-stable-memory libraries provide high-level structures (BTreeMap, Vec, etc.) that make using stable memory ergonomic.
Work is in development for Motoko to use stable memory as the main heap.
Comments from the community:
- Worries about the ability to change schema of data in stable memory (HotOrNot). There are tricks for doing this, but maybe the libraries should make it as easy as possible, or come with examples of how it can be done?
- The size limits for data in the ic-stable-structures lib are a problem which should be removed (HotOrNot).
- More guidance/patterns on using stable memory in Motoko would be helpful (Origyn)
- We should fix the 10% Wasm-native stable memory regression when reading large chunks (Origyn/Portal)
- Canister forking or similar could be helpful for sharding data that no longer fits in a single canister (Byron)
- Generally more examples of non-toy uses of stable memory would be helpful (including data evolution).
- We need the ability to free stable memory
- Tooling for testing with large stable memory could be helpful, since it takes a lot of time to load dummy data into a test canister (Origyn)
Scalability
When developing applications for the IC, multiple scalability limits when choosing between a single and a multi-canister architecture. Examples are: i) limits on each canister’s storage capacity (48 GiB), ii) the number of canisters supported per subnetwork (~100k canisters) and iii) the throughput of calls (~970 updates/s per subnet, ~4000 queries/s per node).
Several opportunities for increasing the IC’s scalability have been identified and rolled out since launch. Examples are: i) improvements in the memory subsystem to achieve higher concurrency for update execution ii) composite queries to allow calls to other canisters at the speed of query calls and iii) deterministic time splicing to allow long running messages, which permits more complex application logic in canisters.
Discussions led to the following takeaways:
- The IC has currently been optimized for stability rather than performance, as we haven’t been running into any significant performance and scalability limits yet.
- Several ideas for novel architectural improvements have been discussed. Examples are: read-only nodes to scale up query processing, different subnet types (storage, not-geo-replicated ones, different consensus protocol)
- Developers would appreciate it if canisters could be made aware of subnet assignment. At the very least, a way to check if another canister is in the same subnet would be desirable. For example, this would be useful to figure out if canisters are in a different legal jurisdictions, based on which we might want to reject a call.
- Discussions around capability support in the IC and whether they are a useful and easy to use abstraction for a wide audience of IC developers. Would for example help to “group” canisters to applications, so that permissions can easily be shared between all canisters of the same application. Access control is a significant pain point in multi-canister architectures
- “Downtime” of dapps because of checkpointing (every 500 blocks) is big problem for dapps on subnets with a large number of canisters (OpenChat)
Scalability 2 - Multi-canister architectures
In this session, Hot or Not and OpenChat presented their dapp architectures, as well as their reasoning towards these architectures.
Hot or Not
- Canister per user architecture because of logical partitioning and actor model maps nicely to nodes in a social graph.
- Planning for flexible multi-subnet architecture
- Upgrades take 1-2 days because they make a backup in a second subnet
- Use ic-state-machine-tests for testing and test upgrades
OpenChat
- Canister per user and group chat architecture. Have multi-subnet architecture in place.
- UserID is a pair of user canister principal and user principal.
- Use queues and batches for inter-subnet messaging.
- User registration: Decide where to register new users based on the size of the user index in each subnet. Check if a user is an Internet Identity principal to prevent bot users.
- Use peer-js and a signaling server to send messages off-chain as well for lower latency.
Immutable Storage API
Nodes have 10x the storage than currently used by the protocol. We can’t just use the available storage for performance reasons. We need to limit the protocol guarantees for the storage to be able to use more of it. The proposal was immutable blob storage. Write in update calls. Reads only in composite queries. Content is only available through the canister that wrote the blob. Content gets deleted when the canister runs out of cycles.
Discussion led to the following key take-ways:
- Not super urgent for teams, as long as storage costs won’t dramatically be increased soon (Origyn
- More interest in shared storage. Access control is not a big concern and can be added on application layer using encryption (Portal) => Data does not need to be served by a canister
- IPFS integration strategic opportunity because of the pervasiveness in the Web3 ecosystem (Origyn)
- IPFS integration (ICPFS) where nodes are able to pin content
- Boundary Nodes used as general IPFS Gateways. This would also allow better caching and a path to the usage of Boundary Nodes as CDNs
Further feedback from teams related that are related:
- Uploading large amounts of data is very slow (Origyn, Portal)
- WASM deduplication would be valuable for teams with large fleets of the same canister that need to be upgraded regularly (Open Chat, HotOrNot)
Backup and restore
- There’s an overwhelming interest in support for backups.
- Most of the support for snapshotting revolves around the use case of recovering from bad upgrades.
- Having irregular upgrades (last month, last week, last day, last few hours) would be a good cadence.
- There are user-level solutions for backups as well as a protocol level-solution. The community is mostly leaning towards a protocol-level solution.
- There’s interest in forking (i.e. the ability to spin up a snapshot as a separate canister).
- The minimum required on the protocol level is the ability to take snapshots of a canister when it is stopped, and forking from an existing snapshot. Besides that, it would be great to allow downloading and uploading the snapshots.
- There’s interest in a System API call for freeing stable memory.
Internet Computer as CDN
This session had three parts. First, Jesse gave a presentation about Portal. Second, Severin from the SDK Team, gave a presentation about the asset canister and asset certification. Third, Rüdiger from the Boundary Nodes team, gave a presentation about the boundary nodes.
Portal
- Quick presentation explaining the process required to upload a full adaptive bitrate (4 copies in different resolutions) video onto the IC:
- Videos are first stored in a temporary storage until they can be processed and optimized for long term storage and delivery
- Currently the temporary storage is S3, but there were talks about cheap storage solutions e.g. “ICPFS” (IPFS nodes on the IC) which could replace S3 as a temporary storage solution for pre-optimized videos
- Other solutions would be using standard IPFS (maybe not ideal for ownership-gated videos because of open nature of IPFS), or uploading to a canister (expensive and slow uploads makes this solution non-ideal), or using one of the file storage protocols build on top of IPFS such as Arweave, Filecoin or Storj
- After they’re stored in temporary storage at a publicly accessible URL the URL can then be submitted to the Portal transcoding network (L2 on the IC) which orchestrates a fleet of nodes. The nodes are are specialized in transcoding, chunking, and uploading content onto the IC using Ffmpeg and agent-js
- The video content is then processed and uploaded to the IC in a distributed manner over several canisters and subnets to try and reduce any load on a particular canister or subnet
- Once the content has completed uploading to a users channel the Portal validator node validates the upload quality and if acceptable then triggers a payment for the work completed. The validator nodes use staking/slashing to ensure bad actors are discouraged from participating in the protocol
- At this time the original video has been processed and uploaded to the users channel canister and the user can begin using the on-chain content in their own apps/dapps just as they would with a video URL from any traditional web2 video provider
- Videos are first stored in a temporary storage until they can be processed and optimized for long term storage and delivery
- Roadblocks to being on par with web2 services are the number of boundary nodes and caching, or more generally better HTTP response times in geo locations such as Japan and Australia (which have a low concentration of Replica nodes and BNs)
- In terms of price, the IC has the ability of offering a cheaper solution to content delivery vs. traditional CDNs as long as the changes to query charging are reasonable (must be much lower than current CDNs to offset the much higher cost of storage and uploading to the IC), performance continues to increase to achieve parity with traditional CDNs, and storage costs continue to decline as new solutions and optimizations emerge
Asset Canister
Asset certification v2 has recently been introduced. The main improvements over v1 are the ability to certify headers, not only the body, and the possibility of selectively certifying only parts of a response. It is included in asset canisters starting at dfx 0.14.0.
Libraries that make certification easy(er) are in the works: Motoko Server (like express.js but for Motoko) is already released in an initial form, but only supports certification v1 so far. A library for convenient v2 certification is in the works for Rust.
Discussion/Take-Aways:
- Most common source for non-matching certificates are fallen-behind replicas or non-matching system time
- Supporting additional encodings (e.g. brotli) is not hard, but will increase the service worker size, which impacts all dapps
- Streaming should be (partially) possible soon. Range headers are not entirely solved.
- Feature requests
- Access control for asset canister. Not on the roadmap, may be left for community implementations
- Expiration for assets. Not on the asset canister roadmap, may be a better fit for something like Motoko Server
Boundary Nodes
The Boundary Nodes are the Gateway to the IC. They offer two endpoints: (1) the API endpoints and (2) the HTTP endpoints serving the service worker or going through icx-proxy.
The API endpoints are defined in the IC’s interface specification. The Boundary Nodes do not apply any filters on the API endpoints, but enforce very moderate rate-limits to protect the replicas, which will result in a 451 status code. Query responses are cached for one second.
The HTTP endpoints serve the service worker or use icx-proxy to translate HTTP requests into API calls. The Boundary Nodes do perform content filtering on these endpoints according to DFINITY’s code of conduct.
When users access content on the IC, they are first routed to the closest Boundary Node (DNS-based). The Boundary Node then forwards the request to a random replica.
Currently, the team is working on decentralizing the Boundary Nodes by splitting them into two entities, the API Boundary Nodes and the HTTP Gateways.
Discussion/Take-Aways:
- Geographical distribution and number of Boundary Nodes - To provide better performance for users, it would be desirable to have more Boundary Nodes better spread across the entire world. As we decentralize the Boundary Nodes and onboard new Node Providers in other regions, we will most likely also have better geographical coverage for the API Boundary Nodes. (Portal)
- Caching - To provide better performance and reduce the number of queries hitting replicas, Boundary Nodes should provide extensive caching. Caching is on our Roadmap, but needs to wait for improvements in the Boundary Nodes. (Portal)
- Analytics - A canister has no information about the client besides the caller, which in many cases is the anonymous identity. It would be great to provide further information (e.g., IP address, user-agent). We realize the importance and have it on our wishlist. Currently, this is very difficult as information is either not known where it could be added to the payload (e.g., the service worker does not know the client’s IP address) or cannot be added (e.g., the Boundary Node cannot modify the request as it is signed by the caller). (Origyn)
- One-Shot HTTP Upgrades - For analytics, it would be great to enable one-shot upgrades that immediately return (like a query), but allow mutating some state, for example, for statistics. We took note of the request, but currently this is not on our roadmap.
Queries & Query charging
Queries are an important part of the IC: their execution is much faster since they are executed by a single node in a non-replicated fashion and thereby avoid the cost of reaching consensus between nodes.
Queries have recently received multiple improvements. For example, there is now one query input queue per canister instead of a global queue shared by all canisters. This provides fairness and better locality.
There is soon also support for caching queries in the replica. Compared to caching in the boundary nodes, replicas can invalidate the cache whenever the state of a canister has changed and therefore avoid unnecessary periodic eviction of the cache.
Another feature related to query processing is charging for query calls: it increases fairness as available hardware resources of IC nodes are assigned to query processing proportionally to how much a canister is charged.
Discussions led to the following takeaways:
- Manipulating query charges does not seem like an attractive attack vector for potentially malicious node providers. Due to aggregation of query stats of many machines over the period of epochs, it is not possible for malicious replicas to significantly impact cycle charges.
- Appending query statistics to blocks does not affect consensus.
- For protection against bots, the sender IP address might be a useful input to the rate limiter. W3C is working on trust tokens. How about having update calls produce the trust tokens and query calls consume them. Capability-based access control should also be considered for access control.
- It might be useful to produce some metadata during query calls (e.g. how often an NFT has been viewed). A similar approach as aggregating query charges might work for this generalized use case as well.
- Developer-defined instruction counters provided by the system API might also be useful.
- Query chunking would be useful for many developers.
Encryption and Data Privacy
Everyone knows that data on the IC is not fully public like it is on other networks, but there are different views on the levels of existing privacy guarantees. We saw a table distinguishing the privacy levels for different types of IC data (ingress messages, state, key material) for the different entities that handle it (canisters, nodes, network). We saw motivation for why we should want encryption and what are the issues with existing solutions; key management is difficult, especially for server-side encryption. We introduced the vetKeys solution by giving a high-level view of how the scheme would work. We then saw different extensions/use cases for vetKeys, namely, encrypted file storage and retrieval (symmetric keys), E2EE messaging (public keys / IBE) and Timelock / MEV protection (encrypting to the ~future). We then saw the first WIP demo of this feature. There were three canisters shown; a canister stubbing the system API of the new functionality, a canister mock of the front end, and the back end of an encrypted file storage dapp. The demo showed a key derivation, a key derivation with file/message encryption, and decryption.
Discussion led to the following key take-ways:
- There are remaining questions about pricing (similar or lower than tECDSA).
- There is demand for multiple demos and for them to be open sourced.
- Storage overheads are to be considered and depend on the encryption scheme used.
- The most motivated ‘consumers’ of vetKeys right now are user-centric dapps (eg OpenChat), businesses (eg Canistore for GDPR reasons), and node providers (to reduce liability).
- There are interesting use cases bridging software (vetKeys) and hardware (SEV).
Partnerships between the teams
The main discussion point was integration between dapps on the IC, and integration with Open Chat in particular. Most teams would like to use Open Chat as an API, not as an iframe or widget. Having to login to an app and to login into OpenChat inside the app is a no go for UX reasons. Most teams would like to take their chats from dapp to dapp. Internet Identity makes these things hard at the moment.
The second discussion point was about funding infrastructure projects. Infra is about solving (current) inefficiencies of the platform. There’s always the fear of getting eaten by the platform as an infrastructure project. Ideas for funding:
- List of verified builders on the IC. DAOs can airdrop shares to builders or provide discounts (Anvil)
- Social norm to reserve parts of an SNS to the infrastructure builders