Hi all, wanted to share v1 of our indexing system. It’s written in Rust and uses the ic-stable-structures crate. This is going to be part of the Dragginz ORM that we’ll open source at some point (probably about 6 months away i think.) Feel free to criticise, I probably don’t know what I’m doing.
Step 1 : Define the Entity, I’ll show a complicated one…
You can have multiple indexes, they can be on any combination of fields (but not those named as primary/sort keys). There’s one optional argument for now, unique.
The API uses a CompoundKey which is a Vec, for example [“character_id”, “pet_id”]. The database itself uses a SortKey however, so Character=123/Pet=456 so we can do range queries and pull parent and child data in the same call. Thanks @icme
Step 2 : CTOR all the things
Every entity/newtype/set/map/list/whatever in our ORM gets sent via ctor to a schema. This structure is serialized to a file so that the front-end can use it, but wait …
Step 3 : Schema goes into memory via crude hack
This is the awesome bit. Json goes into file via a build script, and we can read it using the include_str! macro and Arc allows us to share a pointer around the ORM.
Step 4 : CRUD
The pre-build Schema puts in an automatic crud endpoint for every entity within a canister. We did think about using a variant that represents every entity but the code got really messy.
Step 5 : The Store
The Store requires two stable memories and has a reference to the Schema which knows what entities have which indexes.
As we may be sending types to the front-end we need to generate candid for them, so all the types (even ones used internally via the ORM are defined in the Schema).
Step 6 : The Candid
All automatically generated by the schema!
Step 7 : The Query
We use a WhereQuery structure that allows us to do pagination, ranges, and most importantly indexes.
Why do we need this? One of the use-cases is a loot table where you have to keep pulling rows using different criteria.
Indexes can get out of date if the schema changes, so the system is designeed to allow dropping and rebuilding of index data.
Step 8 : Testing
Here you can see the API in action. We have to run the tests via an endpoint because stable-structures exists within Wasm.
I will be in Zurich in September to show people what we’ve been working on. Feedback appreciated, I’m cool if I have to write the whole thing from scratch, just want it to be as good as it can be.
