Giving mo:base/Buffer a new home in xtended-collections

Developers Motoko
1

For those migrating from base to core library: Buffer won’t be included in core, so I’ve forked it into a standalone package to avoid forcing base library dependency. You can still reference the base library version, but I thought it would be nice to have it standalone and potentially be updated in the future

New package: xtended-collections on MOPS

Xtended-Collections - DynamicArray

import DynamicArray "mo:xtended-collection/DynamicArray";

let dynamicArray = DynamicArray.DynamicArray<Nat>(3);

// Add elements
dynamicArray.add(1);
dynamicArray.add(2);
dynamicArray.add(3);

let array = DynamicArray.toArray(dynamicArray);

Key changes from mo:base/Buffer:

  • Module renamed to DynamicArray (more accurate description of what it actually is)
  • New buffer() instance method exposes generic buffer interface from the new buffer library (see below)

New Buffer, as an interface

I created a new library buffer on MOPS that is a simple write-only interface for appending data:

public type Buffer<T> = {
  write : (T) -> ();
};

Initially, all my encoding libraries used Buffer.Buffer<T> as the standard. With that going away, I rethought the approach and created something generic like Iter.Iter<T> that works across different collection types.

This lets users choose their preferred implementation - whether DynamicArray, List.List<T>, [T], or custom types.

Usage Examples

With DynamicArray:

let dynamicArray = DynamicArray.DynamicArray<Nat8>(3);
Cbor.toBytesBuffer(dynamicArray.buffer(), cborValue);
let cborBytes = DynamicArray.toArray(dynamicArray);

With List:

let list = List.empty<Nat8>();
Cbor.toBytesBuffer(Buffer.fromList(list), cborValue);
let cborBytes = List.toArray(list);

My hope is that this can be useful to others as a simple interface that can be used, but im open to thoughts

1 Like
2

What’s wrong with importing it from base? Don’t think we need a new package. Isn’t it easier to just depend on base?

It doesn’t make the binary larger just because the package contains other stuff. And if you happen to need any of the other things from base then you can import them without needing another line in mops.toml.

I find it less accurate. Buffer is not a dynamic array. It is a wrapper around Array that wraps the type into an option so that a variable length Array can be simulated. Calling it a dynamic array makes it sound like the underlying data structure was more sophisticated than it is. This can mislead new users about the performance.

Overall, let’s not encourage the use of Buffer further. It is dangerous because its worst-case behaviour can make canisters trip over the instruction limit.

3

Nothing. More of a ‘base is dead, use core’ confusion and the repeated messaging of buffer is not recommended by Dfinity for certain cases.
Want to give it a place to live where it can potentially evolve an live next to other collection types so people can pick and choose which implementation they want for their use case vs just having the default option

I guess agree to disagree. Maybe dynamic array isn’t the best but im not sure if buffer really says much about it. To me buffer is putting in temporary bytes/data into a bucket and dynamic array gives the feeling of an array that isn’t static.

Developers that i have talked to seem to like buffer for certain use cases and plan on using it going forward. If you have any specific changes you would make to the implementation to fix any issues, im open to suggestions.