Motoko 🧬 Module+ Easily upgradable stable memory modules standard

[infu]

While the Motoko language provides a way for us to upgrade our stable memory, for a long time it seemed like something was missing. Developers put third-party modules like BTrees, HashMaps, HTTP servers, ICRC3 Logging, Databases, etc, but then each module provides its own upgrade system or doesn’t provide any at all while leaving it to the developers using these to figure things out.

We came up with the following standard which if adopted by library developers will improve developer experience a lot.

There seem to be two types of upgrades.

1. Module types upgrade - A new version of the module needs to upgrade its stable memory. For example, If this is an ICRC3 logging library and it needs to change its data structure. Developers can also write their canisters as multiple modules and changes in their memory will also be such.

2. Custom types upgrade - The module version stays the same, but users need to change their types. For example when we make changes in these types HashMap<Principal, Profile>

Module+ is expanding on this pattern which we called Class+ Writing Motoko stable libraries

Let’s start with how things will look from the perspective of a developer using Module+ libraries.

Notice that we are not doing

let mem_avatars_1 = HashMap.new<Text, Nat>();

If we do that, whenever the HashMap library changes version and its stable memory type is different, our mem_avatars_1 will have a different type. We need to be able to fix the type no matter how the library changed and that’s why we’ve added (.Mem.HashMap.V1) which is part of the standard.

Let’s see what happens if the HashMap library needs a ‘Module types upgrade’

It’s a one-step upgrade process in which mem_avatars_1 will be upgraded to mem_avatars_2 and then mem_avatars_1 will be emptied.
The class avatars has to now point to the new memory.
If we forget to remove this code in consequent upgrades, nothing bad will happen, it will keep working and wont do any additional upgrading.
We can clean it up by removing mem_avatars_1 and changing upgrade to new

Now let’s perform a “Custom types upgrade”
First we add mem_avatars_3 with the same module types version (V2)
Then we execute ‘upgrade’ given us by the library developers.
Its task is to upgrade mem_avatars_2, place it in mem_avatars_3 and delete the old memory.
To do that it expects us to give it a custom function where we change the types of each item.

If we forget about the code and upgrade the canister, again nothing bad will happen, no upgrades will run. We can clean it up by deleting the upgrade code

If Module+ accepts custom types and stores them in memory, it has to provide such upgrade functions.

Chaining upgrades is also possible. At the end of this mem_avatars_1 and 2 will be empty. Each version only provides upgrades from the previous one.

Nested stable memory module upgrades are also possible and a lot easier to do.

Writing Module+

The module is required to provide “Mem” and then the stable memory types like “One” followed by their versions V1 and V2. It picks the last version and its class only works with it.

Let’s see V1

The import line in production will be import MU "mo:mosup"
(MO)took (S)table (UP)grades
It is pretty simple and has helper functions and types.

The module’s memory is stored inside {var inner : ?M} and that’s how it can get deleted in a one-step process and provide protections when leaving/forgetting the upgrade code while doing additional canister upgrades.

This is memory V2
We have added two more record fields ‘name’ and ‘age’.
Now we also need to write an upgrade function from V1 to V2

One last thing to look at is how “Custom types upgrade” works.
Inside the Hashmap module, we add upgrade functions like this one

In a single canister, we typically include several mops libraries along with locally defined modules to organize the codebase. We’ll be using Module+ to manage all of these.

If library developers adopt such a standard, upgrading will be easier, safer and the developer experience much better.
@skilesare @timo @kpeacock @icme @nomeata @tomijaga @claudio @luc-blaeser

Examples here:
[mosup/mo/entry.mo at master · Neutrinomic/mosup · GitHub]

It looks like the Motoko language team is also working on a language feature that will simplify it a bit and this standard will adapt to it. The language feature however won’t free module developers from having to provide upgrade functions, so I believe it will mostly stay the same. The goal here is to not ask canister developers to write their own ‘module type upgrades’ and to simplify ‘custom type upgrades’

[kpeacock]

I like the end result here, although the syntax it requires is unfortunate