Motoko Base Library Changes

Hello ICP developers, and Happy Holidays!

The Languages Team is currently working on major changes to the Motoko base library. Our goal is to improve the consistency and readability of Motoko canister code for both humans and AI.

We want to involve everyone in the community as much as possible in this process. Below is a link to the GitHub repository where you can view the new API, comment on proposals, or even submit your own improvements:

The best place to provide feedback is the Discussions page, where you can vote and comment on changes to the base library. Your opinions are highly valuable, especially right now while we are most flexible with the API design.

If you have any questions about this base library initiative, please feel free to reply here or open a new discussion on the GitHub repository.

Cheers!
~ Ryan

15 Likes

Hey @rvanasa!

What are the main directions/changes you have in mind for the new base library?

2 Likes

Here is a quick overview of the improvements which we have talked about internally:

  • Defining a uniform API for all data structures that can be used in stable memory.
  • Providing mutable and immutable variations of each data structure.
  • Simplifying usage of the Random module and including built-in pseudorandom algorithms.
  • Fixing inconsistencies in the names of functions (e.g. replacing ā€œArrayMutā€ with ā€œVarArrayā€).
  • Replacing the Iter.range() function with Nat.range(), Int.range(), etc. with an exclusive upper bound.

Since this will involve many small breaking changes, the plan is to do everything in one major update. We will provide a migration guide for the new version of the base library, and the original version will still be available for a gradual transition.

For everyone reading this, we want to make sure that these changes will benefit you, so definitely let us know if you have any feedback!

4 Likes

Why do we need breaking changes? Canā€™t the existing methods stay there?

What are the advantages of rewriting things to stable memory with the new orthogonal persistence memory model upcoming? (Because it is more expensive I typically avoid it if possible). I understand it has become the goto in rust but that seems like a rust problem.

a man in a red shirt says don t you put that evil on me

Data structures that are viable in production-real-world applications are unlikely to be using the data structures in base anyway. They typically need more nuance than theyā€™ve typically provided. Perhaps we get the objects we need, but there are already a number of community produced libraries that DFINITY could audit and enhance that might be a better starting point.

3 Likes

The overall goal of these changes is to reduce the learning curve for new developers and improve the ability for AI language models to generate high-quality Motoko code.

Why do we need breaking changes? Canā€™t the existing methods stay there?

Essentially, we want to rework the function names from the ground up so that the base library is more intuitive when coming from mainstream programming languages such as JavaScript and Python.

What are the advantages of rewriting things to stable memory with the new orthogonal persistence memory model upcoming?

Since the base library currently has a mix of different data structure APIs (mutable / immutable, class-based / module-based, etc.), we want to establish a consistent pattern so that data structures are more or less interchangeable with each other for common use cases. We are also planning ahead for the stable classes proposal, although itā€™s worth mentioning that we donā€™t have an official timeline for this feature as of writing this post.

Data structures that are viable in production-real-world applications are unlikely to be using the data structures in base anyway.

Right; agreed. The opinion within the Languages team is that we want the base library to focus on simple, readable ā€œbuilding-blockā€ data structures and defer to third-party packages for specialized or optimized implementations with usability tradeoffs.

4 Likes

Just to clarify, we donā€™t intend to implement new low-level data structures in stable memory directly (e.g. the equivalent of Rustā€™s ā€œstable structuresā€). Instead, we just want to ensure that the Motoko native heap data structures are stable where possible and can easily be stored in stable variables without requiring awkward and dangerous pre-upgrade hooks.

4 Likes

Okā€¦so stable but not stable storage(this has always been awkward). I typically refer to these as sharable, because they can be passed by actors, but I donā€™t think that is quite right either since some functions currently can be passed but arenā€™t ā€œstableā€.

Iā€™d love someone from the team to take a look at the class plus component I just put out. Maybe there is something there that could be further abstracted into the language(and maybe I made some bad assumptions), but I think that it solves a bunch of this from a software viewpoint. When paired with the migration pattern pre and post upgrade goes away. Devs can already stress less about what is stable or not if the lib devs implement it correctly.

https://mops.one/class-plus

3 Likes

Wow, this looks awesome and I canā€™t wait for it and itā€™s obviously needed, but it invokes a sense of existential dread as I realize that days and days of my past life spent in development time are going to be abstracted away into the language.

This is great in the future. But how do other developers deal with this psychologically? I guess itā€™s healthy for any new language, but it ā€œfeelsā€ bad.

1 Like

Thank you for sharing your perspective. I understand where youā€™re coming from with this and want to make sure that we keep the value of the immense amount of work thatā€™s built on the current version of the language. My hope is that stable classes would give more expressiveness to Motoko and complement rather than replace the existing design patterns.

2 Likes

Iā€™m always excited when it comes to introducing changes, especially ones that bring consistency and usability improvements. I really hope the proposed updates introduce more uniformity with module-based data structures and pave the way for shareable data structures. Making it easier to serialize types into stable memory, especially custom ones, and simplifying conversions between types would be a welcome change! Stable classes would be awesome.

On my personal wishlist, Iā€™d love to see a Cycles.measure(func : () -> ()) method to measure the cycles cost of specific operations similar to the countInstructions available in the ExperimentalInternetComputer library. That kind of feature could provide real insights into performance and help developers optimize their code. Another idea would be something like a fabricate transaction feature, similar to dfx ledger fabricate-cycles. Expanding this concept to support fabricating the receipt or sending of ICP (e.g., dfx ledger fabricate-icp) could be great for local testing. Though I know this might fall more under the SDK team than the Motoko team, itā€™s something I wanted to throw out there.

I also think it should be much easier to interact with management canisters like cycles minting and the IC ledger. We shouldnā€™t have to explicitly tell Motoko what the management canister, the ICP ledger, or the cycles management isā€”this is Motokoā€™s platform, it should know by design! It could also abstract away the underlying inter-canister calls to making it easier for developers to use. For example, we could have methods like IC.create_canister(settings : Args) -> CanisterId, Ledger.icrc1_transfer(x : TransferArgs) -> TransferResult, or CMC.notify_top_up(y : NotifyTopUpArg) -> NotifyTopUpResult.

These kinds of abstractions would make development far more intuitive and would be a worthy addition to the base library. Iā€™m excited to see what other changes can be made. How do you plan to manage migration? How long will the original version of the base library remain available? Will there be support for older canisters that rely on the existing base library methods, or will developers need to refactor and redeploy their canisters? Will the base library adopt versioning to allow canisters to specify which version of the library they depend on?

1 Like

Thanks for the suggestions! I like the idea of adding wrapper functions for the management canister, ledger, etc. and will run this past the rest of the team.

How do you plan to manage migration?

The current plan is to nest the original base library within a subdirectory, e.g. mo:base/deprecated/Array. We will also create a migration guide which gives an overview of how to transition to the new modules and functions where relevant.

Will there be support for older canisters that rely on the existing base library methods, or will developers need to refactor and redeploy their canisters?

It will be possible to use the previous base library version during a transitional phase, and afterwards you could downgrade the compiler version (currently 0.13.5).

Will the base library adopt versioning to allow canisters to specify which version of the library they depend on?

We will continue to ship the base library separately from the compiler as a Mops package for flexibility with versioning.

4 Likes

Any update on the suggestions from the team?

Also it slipped me in my initial comment but handling HTTP outcalls needs to be included in the library and should be simplified. When you compare Python (webserver) and Rust (single-threaded server), the experience is vastly different because their libraries provide a high level of abstraction(especially python).

There are libraries on Mops like http-server, server, and ic-websocket-cdk, but they are all quite advanced. I understand the constraints of Motoko, given its isolated execution environment, but there should be a more approachable way for developers to learn and use HTTP outcalls.

Since HTTP outcalls are a breakthrough feature for canisters, having a native API in Motoko would be a valuable addition.

1 Like

I would like to see a major overhaul of the motoko base Array methods. Basic things that exist in other languages are not there, particularly JavaScript has nice methods like: push, pop, shift, unshift, splice, includes, reduce, some and every. The current Array lib is severely lacking in this regard and even using the Append method came with a warning for years.

And someone better not reply ā€œJust use Bufferā€ or something else :smile: I just want arrays to work like arrays do in other languages.

2 Likes

The Deque data structure works stack/queue logic but it doesnā€™t make sense because it returns the structure after performing an operation which can be annoying to work with. The append method has been deprecated for a long time and I am not sure why as its a fairly common occurrence to append new data to arrays. For some, for all and contains exists with buffer but not in the array or hashmap libraries.

1 Like

Could you expand on

Providing mutable and immutable variations of each data structure.

Is this done via new syntax, i.e., is it immediately visible whether a ā€œvariableā€ is mutable or not?

1 Like

Just tell yourself you shouldnā€™t be using anything in base in production unless you specifically know why you are doing so. If you need arrays for storage use Vector from mops. If you need a quick transient list base Buffer from base still seems to be the fastest. But yesā€¦having a base is confusing when so much of it is not production ready and/or memory optimized. I wish it was called ā€˜basicā€™ or ā€˜starterā€™ or ā€˜simpleā€™.

2 Likes

Earlier today, we talked about the idea of wrapping the management canister into the base library, and the decision was to defer this to separate Mops packages. The reasoning is that the version of the management / ledger canisters can change independently of Motoko, which could make things tricky for developers using previous versions of the compiler.

At the moment, itā€™s possible to do this with @ZenVoichā€™s ic and ledger-types Mops packages. This import syntax is also available as another way to access the management canister.

This is also a great point and seems reasonable to include in the base library. I canā€™t make any promises about it happening in the next version, but Iā€™ll at least make sure itā€™s included in the backlog so we can find a good solution for this.

I agree, and we are including a List data structure with similar functionality to lists in Python, Java, C#, and vectors in C++ / Rust. JavaScriptā€™s array implementation is a bit of an outlier (since itā€™s usually a list behind the scenes rather than a true array), but we are doing our best to make the API as familiar as we can for JS developers.

In the next base library version, we are adding a mutable deque (mo:base/Queue) while keeping the purely functional API available by importing mo:base/immutable/Queue. This pattern will also apply to all of the other data structures in the base library.

@michael-weigelt, does this answer your question? Otherwise, let me know and Iā€™d be happy to go into further detail.

3 Likes

I would like to see has or includes that just take an item and donā€™t require a predicate function:

Array.includes(ar, 123)
// or
ar.includes(123)

instead of

Array.find(ar, func(item : Nat) = item == 123) != null

Sadly this is not possible with the current motoko compiler(

1 Like

We are evaluating a few alternative language features which would make it possible to implement these sorts of functions. One option is to include a modified version of Rust traits (equivalent to Haskell typeclasses). Another possibility is to support ā€œmultiple dispatchā€ similar to the Julia language.

For those who are curious, this is related to the expression problem, which shows up in different ways depending on the tradeoffs chosen for a particular language.

This is unlikely to be available for the base library changes in this thread, so we are planning ahead in anticipation of adding functions such as includes() and eventually removing the need to pass equality / comparison functions to data structure operations. I will share more details once we decide on a specific course of action.

2 Likes

Just putting this here to establish a cross reference. Seems like the same concept.