The Languages Team is currently working on major changes to the Motoko base library. Our goal is to improve the consistency and readability of Motoko canister code for both humans and AI.
We want to involve everyone in the community as much as possible in this process. Below is a link to the GitHub repository where you can view the new API, comment on proposals, or even submit your own improvements:
The best place to provide feedback is the Discussions page, where you can vote and comment on changes to the base library. Your opinions are highly valuable, especially right now while we are most flexible with the API design.
If you have any questions about this base library initiative, please feel free to reply here or open a new discussion on the GitHub repository.
Here is a quick overview of the improvements which we have talked about internally:
Defining a uniform API for all data structures that can be used in stable memory.
Providing mutable and immutable variations of each data structure.
Simplifying usage of the Random module and including built-in pseudorandom algorithms.
Fixing inconsistencies in the names of functions (e.g. replacing āArrayMutā with āVarArrayā).
Replacing the Iter.range() function with Nat.range(), Int.range(), etc. with an exclusive upper bound.
Since this will involve many small breaking changes, the plan is to do everything in one major update. We will provide a migration guide for the new version of the base library, and the original version will still be available for a gradual transition.
For everyone reading this, we want to make sure that these changes will benefit you, so definitely let us know if you have any feedback!
Why do we need breaking changes? Canāt the existing methods stay there?
What are the advantages of rewriting things to stable memory with the new orthogonal persistence memory model upcoming? (Because it is more expensive I typically avoid it if possible). I understand it has become the goto in rust but that seems like a rust problem.
Data structures that are viable in production-real-world applications are unlikely to be using the data structures in base anyway. They typically need more nuance than theyāve typically provided. Perhaps we get the objects we need, but there are already a number of community produced libraries that DFINITY could audit and enhance that might be a better starting point.
The overall goal of these changes is to reduce the learning curve for new developers and improve the ability for AI language models to generate high-quality Motoko code.
Why do we need breaking changes? Canāt the existing methods stay there?
Essentially, we want to rework the function names from the ground up so that the base library is more intuitive when coming from mainstream programming languages such as JavaScript and Python.
What are the advantages of rewriting things to stable memory with the new orthogonal persistence memory model upcoming?
Since the base library currently has a mix of different data structure APIs (mutable / immutable, class-based / module-based, etc.), we want to establish a consistent pattern so that data structures are more or less interchangeable with each other for common use cases. We are also planning ahead for the stable classes proposal, although itās worth mentioning that we donāt have an official timeline for this feature as of writing this post.
Data structures that are viable in production-real-world applications are unlikely to be using the data structures in base anyway.
Right; agreed. The opinion within the Languages team is that we want the base library to focus on simple, readable ābuilding-blockā data structures and defer to third-party packages for specialized or optimized implementations with usability tradeoffs.
Just to clarify, we donāt intend to implement new low-level data structures in stable memory directly (e.g. the equivalent of Rustās āstable structuresā). Instead, we just want to ensure that the Motoko native heap data structures are stable where possible and can easily be stored in stable variables without requiring awkward and dangerous pre-upgrade hooks.
Okā¦so stable but not stable storage(this has always been awkward). I typically refer to these as sharable, because they can be passed by actors, but I donāt think that is quite right either since some functions currently can be passed but arenāt āstableā.
Iād love someone from the team to take a look at the class plus component I just put out. Maybe there is something there that could be further abstracted into the language(and maybe I made some bad assumptions), but I think that it solves a bunch of this from a software viewpoint. When paired with the migration pattern pre and post upgrade goes away. Devs can already stress less about what is stable or not if the lib devs implement it correctly.
Wow, this looks awesome and I canāt wait for it and itās obviously needed, but it invokes a sense of existential dread as I realize that days and days of my past life spent in development time are going to be abstracted away into the language.
This is great in the future. But how do other developers deal with this psychologically? I guess itās healthy for any new language, but it āfeelsā bad.
Thank you for sharing your perspective. I understand where youāre coming from with this and want to make sure that we keep the value of the immense amount of work thatās built on the current version of the language. My hope is that stable classes would give more expressiveness to Motoko and complement rather than replace the existing design patterns.
Iām always excited when it comes to introducing changes, especially ones that bring consistency and usability improvements. I really hope the proposed updates introduce more uniformity with module-based data structures and pave the way for shareable data structures. Making it easier to serialize types into stable memory, especially custom ones, and simplifying conversions between types would be a welcome change! Stable classes would be awesome.
On my personal wishlist, Iād love to see a Cycles.measure(func : () -> ()) method to measure the cycles cost of specific operations similar to the countInstructions available in the ExperimentalInternetComputer library. That kind of feature could provide real insights into performance and help developers optimize their code. Another idea would be something like a fabricate transaction feature, similar to dfx ledger fabricate-cycles. Expanding this concept to support fabricating the receipt or sending of ICP (e.g., dfx ledger fabricate-icp) could be great for local testing. Though I know this might fall more under the SDK team than the Motoko team, itās something I wanted to throw out there.
I also think it should be much easier to interact with management canisters like cycles minting and the IC ledger. We shouldnāt have to explicitly tell Motoko what the management canister, the ICP ledger, or the cycles management isāthis is Motokoās platform, it should know by design! It could also abstract away the underlying inter-canister calls to making it easier for developers to use. For example, we could have methods like IC.create_canister(settings : Args) -> CanisterId, Ledger.icrc1_transfer(x : TransferArgs) -> TransferResult, or CMC.notify_top_up(y : NotifyTopUpArg) -> NotifyTopUpResult.
These kinds of abstractions would make development far more intuitive and would be a worthy addition to the base library. Iām excited to see what other changes can be made. How do you plan to manage migration? How long will the original version of the base library remain available? Will there be support for older canisters that rely on the existing base library methods, or will developers need to refactor and redeploy their canisters? Will the base library adopt versioning to allow canisters to specify which version of the library they depend on?
Thanks for the suggestions! I like the idea of adding wrapper functions for the management canister, ledger, etc. and will run this past the rest of the team.
How do you plan to manage migration?
The current plan is to nest the original base library within a subdirectory, e.g. mo:base/deprecated/Array. We will also create a migration guide which gives an overview of how to transition to the new modules and functions where relevant.
Will there be support for older canisters that rely on the existing base library methods, or will developers need to refactor and redeploy their canisters?
It will be possible to use the previous base library version during a transitional phase, and afterwards you could downgrade the compiler version (currently 0.13.5).
Will the base library adopt versioning to allow canisters to specify which version of the library they depend on?
We will continue to ship the base library separately from the compiler as a Mops package for flexibility with versioning.
Also it slipped me in my initial comment but handling HTTP outcalls needs to be included in the library and should be simplified. When you compare Python (webserver) and Rust (single-threaded server), the experience is vastly different because their libraries provide a high level of abstraction(especially python).
There are libraries on Mops like http-server, server, and ic-websocket-cdk, but they are all quite advanced. I understand the constraints of Motoko, given its isolated execution environment, but there should be a more approachable way for developers to learn and use HTTP outcalls.
Since HTTP outcalls are a breakthrough feature for canisters, having a native API in Motoko would be a valuable addition.
I would like to see a major overhaul of the motoko base Array methods. Basic things that exist in other languages are not there, particularly JavaScript has nice methods like: push, pop, shift, unshift, splice, includes, reduce, some and every. The current Array lib is severely lacking in this regard and even using the Append method came with a warning for years.
And someone better not reply āJust use Bufferā or something else I just want arrays to work like arrays do in other languages.
The Deque data structure works stack/queue logic but it doesnāt make sense because it returns the structure after performing an operation which can be annoying to work with. The append method has been deprecated for a long time and I am not sure why as its a fairly common occurrence to append new data to arrays. For some, for all and contains exists with buffer but not in the array or hashmap libraries.
Just tell yourself you shouldnāt be using anything in base in production unless you specifically know why you are doing so. If you need arrays for storage use Vector from mops. If you need a quick transient list base Buffer from base still seems to be the fastest. But yesā¦having a base is confusing when so much of it is not production ready and/or memory optimized. I wish it was called ābasicā or āstarterā or āsimpleā.
Earlier today, we talked about the idea of wrapping the management canister into the base library, and the decision was to defer this to separate Mops packages. The reasoning is that the version of the management / ledger canisters can change independently of Motoko, which could make things tricky for developers using previous versions of the compiler.
At the moment, itās possible to do this with @ZenVoichās ic and ledger-types Mops packages. This import syntax is also available as another way to access the management canister.
This is also a great point and seems reasonable to include in the base library. I canāt make any promises about it happening in the next version, but Iāll at least make sure itās included in the backlog so we can find a good solution for this.
I agree, and we are including a List data structure with similar functionality to lists in Python, Java, C#, and vectors in C++ / Rust. JavaScriptās array implementation is a bit of an outlier (since itās usually a list behind the scenes rather than a true array), but we are doing our best to make the API as familiar as we can for JS developers.
In the next base library version, we are adding a mutable deque (mo:base/Queue) while keeping the purely functional API available by importing mo:base/immutable/Queue. This pattern will also apply to all of the other data structures in the base library.
@michael-weigelt, does this answer your question? Otherwise, let me know and Iād be happy to go into further detail.
We are evaluating a few alternative language features which would make it possible to implement these sorts of functions. One option is to include a modified version of Rust traits (equivalent to Haskell typeclasses). Another possibility is to support āmultiple dispatchā similar to the Julia language.
For those who are curious, this is related to the expression problem, which shows up in different ways depending on the tradeoffs chosen for a particular language.
This is unlikely to be available for the base library changes in this thread, so we are planning ahead in anticipation of adding functions such as includes() and eventually removing the need to pass equality / comparison functions to data structure operations. I will share more details once we decide on a specific course of action.