Motoko Stable Regions

Developers
1

The Motoko team has an experimental branch of the compiler that implements stable regions.

For context, Motoko currently exposes low-level canister stable memory through an experimental API.

For those developers who implement or use existing stable data structures, I’m very curious to know what you think of the new API, which is a minimal generalization over the existing API.

For instance, here are the small changes needed to adapt this data structure, I expect:

For some context, see this part of the forum thread about the structure.

What is a “region”?

In essence, a Region is a new abstract data type in Motoko that admits the same API as before, except that distinct regions use distinct portions of stable memory, and never interfere with one another.

Critically, new Regions can also be created dynamically, and old ones that become “garbage” can (eventually, in subsequent revisions of this system) be collected and their portion of stable memory reused for a new Regions, in the future. The current WIP branch implements dynamic allocation, and provisions in the design will eventually permit automatic reclamation.

For now, the old API still “works”, but it uses a special, pre-allocated region, isolated from all others created dynamically. We will use this special region as a migration path (WIP now) for existing stable memory structures that were created and maintained using the current, experimental API.

What can we do now?

Let me know if you have a stable data structure that I can try to adapt, like the example above!

If you want access to the branch of the compiler and base library that permit you to do your own tests, let’s discuss that. It should be possible.

Another example

As another example of using the new Region type, see this WIP PR in base that gives a “stable buffer” that permits growth with arbitrary-sized blobs, and random-access reads based on numerical indicies..

Caveat: I’m not sure if this belongs in base, but it’s a place to have the PR for now.

What’s left to do?

Before merging this feature:

  • Misc cleanup of the PR
  • More tests (basic tests show that the mechanism works; we need more though)
  • Migration path, plus tests for it (installing the region manager and creating “region zero” for existing data).
14 Likes
2

The PR is finished, and is under review now.

While that review happens, we encourage early adopters to try moc with this feature now.

(The PR has an moc artifact, and by giving it --stable-regions as an argument, folks can try the new region system before it lands in branch master.)

Please reach out if you have issues using the feature, or have any questions about it.

cc @skilesare

(Update: Now both Mac and Linux binaries are built by CI. Initially, there was only a Linux binary.)

3 Likes
3

Update: Last week, this feature was finally merged into the main compiler branch :tada:

(Consequently, it will be included in the next version of DFX.)

In the meantime, the release page has a pre-built compiler for each release, and each platform.

5 Likes
4

Nice work!

Are there any plans to include at least 1-2 new stable data structures (that use Region) in base, so that developers can see how to best build libraries around a Region, as well as learn preferred usage patterns and edge cases to consider when building on top of Region?

1 Like
5

Good question. I’d like to include at least one map implementation, and perhaps also an implementation of sequences that grow by appending (suitable for very long logs).

During this work, I had planned on adapting the work of @sardariuss on the “stable B tree” Bounty. However, this message is giving me some second thoughts:

I’m curious to hear your thoughts @icme

6

This message also made me think about using this library. I’ve had some bad experiences in a local environment where the replica would stop responding after 400 mb of data in stable memory. I didn’t understand what the reason was, but the conclusion was disappointing - while rust developers have excellent stable structures with support from developers from the DFINITY team, motoko developers have one library, the author of which does not recommend using it.

2 Likes
7

The purpose of the bounty you’re mentioning was to create a Motoko version (parity) of the BTree in

This library is maintained by Rust engineers at DFINITY, so it would be very helpful to the Motoko community if at a BTree version of this (at least) was available for developers. As an aside, one of the main friction points developers run into when using the library is max key/value size bound limits.

All this being said, I know there’s also the potential for all of this to be abstracted away from Motoko developers in the future by some of @luc-blaeser’s work, given the improvements to wasm-native stable memory. This would be the ideal solution and would showcase Motoko’s superpowers as an IC native language, but I’m not sure how far away this is from becoming a reality in terms of the language’s roadmap.

In the meantime, in order of preference these are what I’d imagine are the most important data structures to have stable memory libraries for.

  1. Map (BTree recommended)
  2. Array (could be abstractly built on top of BTree)
3 Likes
8

Sorry about that, I understand your frustration, but it was just my honest opinion at that time. Moreover I think a library like this requires at least some code reviews before being used in production. Maybe ICdev did some but i got no feedback at that time.

For the issues you mentionned I’ve been doing some investigations in the past but i couldnt find out the root cause. I am gonna give it a fresh look at it again, and if I manage to make more time (i.e. money :sweat_smile:) i’ll try to implement the new stable regions, it looks quite sexy.

3 Likes
9

If anyone is online now and wants to chat about this, or any other Motoko thing, we (the Motoko team) are online for the next 50min or so in this Zoom:

Please feel free to join, if only to listen.

10

I understand regions can grow but not shrink.

Can they be entirely deleted? What happens if I “forget” the reference to a region, does that result in its deletion?

2 Likes
11

I would like Stable Memory to have “shrink” feature too. For now it’s only half-scalable since it can only “grow”.

1 Like
12

I found a remark in an article here saying " any old regions that are marked for garbage collection will be removed and reused for new Region s afterwards". So it seems that regions can indeed be deleted in this way by dereferencing them.

It’s not clear though how usable that is. Say I have a large region which dominated my canister’s memory footprint, dereference it, then create a new large region. If the garbage collection does not run in between the two events then I accidentally double my canister’s memory footprint and will pay storage cost for the doubled memory forever.

13

Just to be clear, garbage collection of the stable memory pages used by an unreachable region is not implemented as of 0.10.1. While Region objects do get collected, the stable memory pages they hold on to do not.

See Shrink Region by using Region.grow(region, 0) - #4 by claudio