Pre-release `moc` 0.14.0 available in Playground

Hello Motokoders,

I am happy to announce the availability of the Motoko compiler version 0.14.0 (a pre-release so far) in the Motoko Playground! Please feel free to experiment and take this out for a ride. We’ll let it soak a bit and solicit bug reports in our issue tracker. When we feel happy with it, we’ll promote it towards dfx too. In the meantime you can also include it into your current setup using the DFX_MOC_PATH envvar.

Big kudos to @claudio for implementing this feature.

In brief, this version introduces a clean way of attaching explicit data migration functions to actors (and actor classes). There are several guardrails in place to prevent data loss, and the look-and-feel is largely improved in comparison to the low-level system pre/postupgrade hooks.

Please check out the release notes and the examples in the documentation.

Have fun!

— The Motoko team

Update: I added a better link above.

The docs don’t render nicely from the PR, anyway here is one of the example sources, you have to browse around a bit:

Sorry for the inconvenience, this will be nicer when deployed as proper documentation…

Can this be applied “down the stack” and/or can it handle multiple different objects. Most actors out there are an amalgamation of various classes each with their own state and migration cadence.

Basically, I want to put the migration in the class and have the actor builder not have to worry about it.(otherwise we rapidly reduce composeability.

The top-level actor is monolithic w.r.t. stable bindings. (We are thinking about actor modules, but it is not a priority/possibility mid-term). So the migration function must somehow also be monolithic. Of course you can put the data migration into a module and use the record slicing property to pass subsets of old attributes along different sub-migrations, collecting up the results like { subMigrA(oldAttrs) and subMigrB(oldAttrs) and ... } (i.e. record concatenation) and handing them back as the new stable bindings. subMigrA could receive disjoint record fields from subMigrB, but there can be overlaps. So you can do everything in a modularised way. But the main spider must be attached to the main actor.

Maybe I am interpreting “down the stack” incorrectly, an example would surely help.

Addendum: I fleshed this suggestion out as a simplistic compilable example.

Ah, maybe you are hinting at actor classes in a canister whose instances you want to upgrade to new revisions programmatically? You can attach parenthetical notes to the actor classes too!
Same rules apply. Each actor class can have its own (with migration = ...).

Using that syntax I guess I was hoping for something like this:

import TT from "mo:TT";
import ICRC72Subscriber from "mo:subscriber";
import ICRC72Publisher from "mo:publisher";



(with migration = func(old : OldStables) : NewStables = { 
  TT.Migrations.migrate(old) and 
  ICRC72Subscriber.Migratons.migrate(old) and 
  ICRC72Publisher.Migrations.migrate(old) 
})
persistent actor {
  stable var tt_migration_state: TT.State = TT.Migration.migration.initialState;
  stable var icrc72SubscriberMigrationState : ICRC72Subscriber.State = ICRC72Subscriber.Migration.migration.initialState;
  stable var icrc72PublisherMigrationState : ICRC72Publisher.State = ICRC72Publisher.Migration.migration.initialState;
};

My concern with this:

type OldStables = { a : Nat; b : Char; c : Blob };
type NewStables = { g : Nat32; h : Text; i : Char };

/// ^^^ above declarations should go into modules ^^^

Is that TT will have no understanding of ICRC72Subscriber and thus won’t be able to declare the type? I’m guessing old will throw a type exception because it will have so many more member than expected.

Maybe this would work?

import TT from "mo:TT";
import ICRC72Subscriber from "mo:subscriber";
import ICRC72Publisher from "mo:publisher";

(with migration = func(old : OldStables) : NewStables = { 
  TT.Migrations.migrate({old.tt_migration_state}) and 
  ICRC72Subscriber.Migratons.migrate({old.icrc72SubscriberMigrationState}) and 
  ICRC72Publisher.Migrations.migrate({old.icrc72PublisherMigrationState}) 
})
persistent actor {
  stable var tt_migration_state: TT.State = TT.Migration.migration.initialState;
  stable var icrc72SubscriberMigrationState : ICRC72Subscriber.State = ICRC72Subscriber.Migration.migration.initialState;
  stable var icrc72PublisherMigrationState : ICRC72Publisher.State = ICRC72Publisher.Migration.migration.initialState;
};
1 Like

What I’d really like to do would be more like

import TT from "mo:TT";
import ICRC72Subscriber from "mo:subscriber";
import ICRC72Publisher from "mo:publisher";

(with TT.Migration.migrate, ICRC72Subscriber.Migration.migrate, ICRC72Publisher.Migration.migrate) //or an and between them or something...or maybe an array of them)
})
persistent actor {
  stable var tt_migration_state: TT.State = TT.Migration.migration.initialState;
  stable var icrc72SubscriberMigrationState : ICRC72Subscriber.State = ICRC72Subscriber.Migration.migration.initialState;
  stable var icrc72PublisherMigrationState : ICRC72Publisher.State = ICRC72Publisher.Migration.migration.initialState;
};

Or even better, put the migration function in my class so that it “just work” with Orthogonal persistence. We currently handle migration inside the class with a pattern that can be seen here: icrc72-subscriber-mo/src/lib.mo at d16fd91c8b824ac6aafffcac300311876d9d1de5 · PanIndustrial-Org/icrc72-subscriber-mo · GitHub (See line 172…changes are made in the class instantiation by making changes to the passed in “memory”)

You can always use object type concatenation and have something like

migration = func(old : TT.Old and ICRC72Subscriber.Old and ... ) : TT.Curr and ICRC72Subscriber.Curr and ... = ...

as long the record fields are compatible (or even better, don’t overlap).

But why not make a top-level migration module where you aggregate the constituent migration pieces?

1 Like

That is very risky in the general case. Since old and new stable bindings are all in scope, nasty circular uses can be written (staging violations), and making it safe by compiler analysis is hard. The chosen scheme excludes such anomalies and is general enough for most use cases. The previously worked out patterns continue to work and you can drop down to the system hooks if needed.

Keep feeding us with examples and we’ll help with recipes.

In theory module A, B, and C are built by different development shops, are audited and provide functionality in different domains. They are prebuilt functionality that a dev can add into their canister with little or, ideally, 0 knowledge of how or why the work. We certainly don’t want the canister developer to have to worry about migration. They should ‘just work’.

For example, I want a canister that is an ICRC-1/2/3/(ICRC-1 module, ICRC-2 Module that requires ICRC-1(but ICRC1-knows nothing of 2), ICRC-3 Module for logging that is completely agnostic to tokens and is just a ledger compatible token that uses ICRC-75 to maintain a list of accounts that are each issued tokens once a month via TimerTool module that is fed events and makes scheduling that survives restart and upgrade as easy as writing a couple of lines of code.

In an ideal world, all can just be wired together(and yes actor modules would be huge here because right now when we do this we have a ton of boilerplate that equates to public shared(msg) my_func(args :MyModule.args){ await* MyModule.myFunc(msg.caller, args)};

import TT "mo:timer-tool";
import ICRC1 "mo:icrc1.mo";
import ICRC2 "mo:icrc2.mo";
import ICRC3 "mo:icrc3.mo";
import ICRC75 "mo:icrc75.mo";

(with migration = func(old: TT.old and ICRC1.old and ICRC2.old and ICRC3.old and ICRC75.old) :
{TT.new and ICRC1.new and ICRC2.new and ICRC3.new and ICRC75.new}{
  {
     ttState = TT.migrate(old.ttState);
     icrc1State = ICRC1.migrate(old.icrc1State);
     icrc2State = ICRC2.migrate(old.icrc2State);
     icrc3State = ICRC2.migrate(old.icrc2State);
     icrc75State = ICRC2.migrate(old.icrc2State);
};
persistent actor class MyToken(args: MyTpes.args)  = this {
   
     //I'm guessing with orthogonal persistence these don't need to be stable anymore?
     var ttState = TT.initialState : TT.new;
     var icrc1State = ICRC1.initialState : ICRC1.new;
     .... other states
     
     //now we pass these states to classes(perhaps we get stateful classes in the future)
     let tt = TT.TT(ttState, {environmentVariables});
     let icrc1 = ICRC1.ICRC1(icrc1State, {environmentVariables and handler functions};
     ....
     
     // ideally the : public shared(msg) (ICRC1.TransferArgs) => async Nat would be optional
     public shared icrc1_transfer = icrc1.transfer : public shared(msg) (ICRC1.TransferArgs) => async Nat;
     .... and so on
     .... but also the following so you can override the default function if desired
     public shared override icrc2_approval......
};


I guess super ideal would be something like inheritance for actor classes so the shared functions get added auto magically and you only have to write some code if you want to override.

If you could put the with Migration in that sub actor then the end user of the component wouldn’t need to know about migration at all…just mark it as persistent and add all the descendant classes and it will look for and call the migration mentioned on any public vars in that class(which I guess should get prompted and ‘injected’ into the wrapping actor as well.

This is how I want to be able to write a canister from multiple third party components…this currently would be probably hundreds of lines of code shorter than what I currently have to do:

import TT "mo:timer-tool";
import ICRC1 "mo:icrc1.mo";
import ICRC2 "mo:icrc2.mo";
import ICRC3 "mo:icrc3.mo";
import ICRC75 "mo:icrc75.mo";

//args would need to be 'smart' here and know which item in args to pass to each class or maybe they are dumb and HAVE to have no arguments and is assumed the user will call something like .init()
//and I don't know about the notation, I know `:` is already types...so I used 
persistent actor class MyToken(args: MyTypes.args) :: {
     TT.TT as tt; 
     ICRC1.ICRC1 as icrc1, 
     ICRC2.ICRC2 as icrc2;
     ICRC3.ICRC3 as icrc3;
     ICRC75.ICRC75 as icrc75;  = this {

    ///define envVars, handlers, config, et
    tt.init(envVars);
    icrc3.init(envVars); //we have to do 3 first because it is passed to 1
    icrc1.init({
      addRecord = icrc3.addRecord;
      symbol = "MYT";
      logo = "data;....";
      name = "My Name"
      ....});
    icrc2.init(envVars);
    ....etc
   
    tt.setTimer(now + one_month, paytokens, null):

    private payTokens(eventArgs: TT.EventArgs){
      for(thisItem in ICRC75.ICRC75.instance.membersOf("com.mylist")){
         ICRC1.ICRC1.instance.tranfer(mintingAccount, thisItem, 1_000_000
      }
    };

    public override shared icrc2_approve(approveArgs: ICRC2.ApproveArgs) : async Nat{
      //custom code for this functions
    };
};


//other file for icrc1 definition...do migrations here.
//type defs

//migration def

(with migration = migrations)
public subactorclass ICRC75(){

    public func init(env: ENV) {//define};

    var accounts = Map.Map<Account, Nat>;

    public shared(msg) function icrc1_transfer(args: TransferArgs) : async Nat {
      //define
    };
    ....other funcs

}
1 Like

I think something like this can be made to work without too much trouble:

A.mo

module  A {

   public type State = {#A1};

   public let init : State = #A1;

   public module Old = {
    public type State = Any // use Any just so we can repeat upgrades
   };

   public func migrate(_old : Old.State) : State = #A1;
 
}

B.mo

module  B {

   public type State = {#B1};

   public let init : State = #B1;

   public module Old = {
    public type State = Any // use Any just so we can repeat upgrades
   };

   public func migrate(_old : Old.State) : State = #B1;
 
}

Migration.mo

import A "A";
import B "B";
import Debug "mo:base/Debug";

module {
  public func migration(
    old : {
      var a : A.Old.State;
      var b : B.Old.State
    }
  ) : {
    var a : A.State;
    var b : B.State
  } {
    Debug.print "migrating";
    {
      var a = A.migrate(old.a);
      var b = B.migrate(old.b)
    }
  }
}

Main.mo

import A "A";
import B "B";

import {migration} "Migration";

(with migration) 
persistent actor {

  var a = A.init;
  var b = B.init;

  public func show() : async Text {
    debug_show {a; b};
  }

}
1 Like

TT will only declare its own concerns:

  • TT’s migration will only take { ttOld1 : TT_typeA; ttOld2 : TT_typeB; ... }
  • and return { ttCurr1 : TT_typeX; ttCurr2 : TT_typeY; ... }.

Even if the old global record contains a lot of other fields, passing those to TT’s migration will expose only the ttOld* to it. This is what I meant by record slicing above.

1 Like

Ahh…great. I understand. This is nice! I can certainly work with this in the short term.

SubActor classes are still on my list though! I think they’ll make actually pulling together a canister much easier and that will make it easier to teach something ‘useful’ and thus a virtuous circle will emerge.

a virtuous circle

1 Like

On mine too! But it is really hard to pull off and needs to deal with a long list of requirements. But in the end it will need to have migration functions in isolation which create just the (subset of) stable bindings that the actor module defines.

1 Like

Does the new migration syntax require enhanced orthogonal persistence, thus wasm64, or is it an independent feature?

And I suppose old and new canisters must both be compiled with moc 0.14.0, or? How can I use this starting out with a moc 0.13.x canister?

No, it works with classical persistence. Only the current canister must be compiled with the 0.14 series, as long as the old one has the right .most metadata. Can you confirm, @claudio?

But you can try it in the Playground or download the release.

1 Like

It works with both eop and classical.
And you should be able to upgrade an old, pre 0.14.0 canister just fine, including with migration. Only the new code has to be compiled with 0.14.0.

Having claimed that, I have not actually tried it.

The thing that won’t work is upgrading from a canister with migration code to a canister compiled using an older compiler, although I expect it should still work if you ignore the dfx warning.

1 Like