Awesome! Glad to see someone take a stab at this. Where were you 2 years ago @frederico02 
So CanDB is really a framework that has two different pieces. A backend library (currently only written in Motoko), and a client (currently only written in TypeScript).
The backend module is composed of two main parts:
- Storage (CanDB)
- Auto-scaling and partitioning module (CanDB Admin)
The storage module gets embedded in partitioned microservice actors, or Service Actors. This module is essentially an opinionated BTree.
The module responsible for auto-scaling and partitioning is embedded into the IndexCanister Actor.
The client module assumes that is working with a CanDB application, and provided the Index Canister Id, wraps query and update calls, abstracting communication between the frontend and service actor parition(s) where the data is stored, so that the client just calls the Index Canister and says call this API in this partition and the framework does the rest.
CanDB does all this in a way that doesn’t require inter-canister calls, which cuts down on latency and is how the original Supernova demo was able to run query and update calls across sharded data so quickly.
As CanDB is a framework meant for performance and low-latency UX with a frontend, it’s primary use case wasn’t originally intended for integration directly with other on-chain canisters. For that, you’d need to build a Motoko/Rust/Azle CanDB client, but inter-canister latency does slow things down considerably (it would still work though, and I’d be happy to advise someone on building additional clients).
I therefore think the primary value add from this grant would be bringing a canister partitioning & microservices framework/pattern to the Rust community.
There are a few key differences between Motoko & Rust that the implementer will need to consider:
- Motoko has the concept of
stable variables, which actually live in the heap, and perform automatic, language-enabled serialization to stable memory during canister upgrades, and deserialization back to the heap afterwards. The CanDB framework makes frequent use of stable variables. The Rust implementation of CanDB may want to swap this out for stable memory use and the stable-structures library. Otherwise, if they want to use heap memory to store the data (for performance reasons or other), they need to build a good abstraction for the serialization and deserialization logic during upgrades.
- The CanDB entity type uses a compound variant type for dynamically storing different data types. The implementer will need to consider how to replicate this in Rust (with the stable/heap memory considerations described in #1).
- CanDB has auto-scaling functionality primarily because the canister heap is currently limited to a memory cap of 4GB (limitation of 32-bit wasm). However, the current canister memory limit is 400GB (entire memory, combination of both heap and stable memory), where stable memory has access to the full 400GB of memory. If the implementer decides to use stable memory Rust stable memory libraries, they can essentially use all 400GB. Additionally, given that the current subnet size is 750GB, does it make sense to turn on auto-scaling if a single canister can essentially be as large as a subnet?
- Given the points made in #3, you may actually want to be able to spin up different partitions and limit the size of each partition, rather than worry too much about auto-scaling.
Given the points made around Rust having easier access to stable memory in #3, I think the canister partitioning scheme that CanDB uses might be the most valuable part to port/adapt over to Rust, coupled with replacing the Motoko stable keyword libraries with stable structures.
There are a number of applications that have this single Index Canister to many Service Actor Canisters pattern from OpenChat to Yral, so it would be nice to see some generic and extensible multi-canister frameworks spin up that would reduce the amount of boilerplate developers need to write if they choose this app architecture.
I don’t want to distract away from the grant objective too much, but I might even recommend taking the bones of CanDB and then talking to @hpeebles and @saikatdas0790 from those respective teams to see what superpowers the Rust CDK has that would work well in a Rust adaptation of CanDB. I know Dragginz has a pretty extensible ORM and DB that @borovan’s been building out as well.
There’s also a number of utility APIs in the CanDBAdmin library that would be super helpful in managing a cluster of service actors. For example, an API that performs a paginated upgrade of all of the service actor canisters in a specific partition key range https://www.candb.canscale.dev/CanDBAdmin.html#upgradeCanistersInPKRange.
So I guess for scope:
- Taking into account feedback/learnings from other Rust multi-canister projects
- port/adaptation of the CanDB module to Rust using stable structures
- port/adaptation of the Index Canister pattern
- No auto-scaling (auto-scaling is less necessary given that Rust canister stable memory is very large).
That way, a developer can build their backend in Rust or Motoko, and the TypeScript client can consume from either without changing the client API.
