We are currently working on creating a NoSQL-type database on the IC using Motoko. Our team has built structures that allow users to create tables, columns, make predefined queries, and create indexes. Users can also create new fields by selecting data types. The goal is to ensure smooth functionality without data migration concerns during canister upgrades.
At the moment, we are utilizing stable memory to achieve these functionalities. We store all the data in a hash map, and during upgrades, we convert it into a stable variable array. This approach allows us to utilize a maximum of 4GB of storage.
Before proceeding with full-fledged development, we would greatly appreciate your guidance on utilizing stable memory instead of stable variables. We are seeking insights on how to maintain the same level of user experience while using stable memory. Any advice or suggestions would be highly valuable.
Awesome to see more database/data structure work on the Motoko side of things! A few resources that might help you along your journey.
If you’re looking for a fairly well tested jumping off point, I’d recommend looking at CanDB. It’s currently used by several large IC applications (including Kinic, which just SNSed) and is backed by the ORIGYN Foundation.
CanDB leverages Motoko’s stable keyword with stable variant types, allowing for Motoko to handle the serialization to stable memory and deserialization from stable memory during canister upgrades instead of requiring the developer to implement pre/postgrade. As you mentioned, using the heap limits per canister memory to 4GB, but you will only get the full 4GB if you use the new incremental GC (you’ll run into instruction limit errors around 1-2GB otherwise.
The performance of stable memory is currently being worked on actively by a few people at DFINITY, so I think it’s great to start building something in that direction now. I’d recommend looking into both this heap based stable BTree and the work from @sardariuss on his BTree that utilizes stable memory directly.
I’m sure that the DFINITY grants team and @domwoe would be willing to fund any Motoko data structure work that lives in stable memory, or that uses the new Motoko Stable Regions API. I’ll also tag a few members of the Motoko team to see if they have any ideas or can provide additional direction @claudio@matthewhammer@luc-blaeser.