Increased Canister Smart Contract Memory

Wowļ¼Œ thank you for your explanation ļ¼šD

1 Like

dfx 0.8.1:
in dfx.json

"build":{
   "args" : "--compacting-gc",
    Ā·Ā·Ā·Ā·Ā·Ā·
}
5 Likes

Update: the implementation of the 64-bit stable memory is part of the new replica release (version e86ac9553a8eddbeffaa29267a216c9554d3a0c6) that will roll out to subnets soon.

13 Likes

I may have missed this, but is this fully rolled out?

Is it supported natively using the stable variable qualifier in Motoko? If not, are there stopgap tools we can use to access memory addresses directly in Motoko?

Thanks!

1 Like

@jzxchiang: youā€™re right. The new version e86ac9553a8eddbeffaa29267a216c9554d3a0c6 with 64-bit stable memory has been rolled out to all subnets as of today.

I will defer the question about Motoko support for @claudio.

5 Likes

Correct. The memory of canister smart contracts has gone from 4 GB to 8 GB memory. No need for developers to do anything.

This is the first step of many on the methodical increase :roller_coaster: towards 300 GB.

4 Likes

Motoko does not yet make use of the 64 bit API, e.g., the amount of data stored in stable variables is currently still limited to 4 GiB. But of course we intend to upgrade soon.

(We also have recently implemented a (still experimental) API for accessing stable memory directly. This will also support 64 bit at the time we release it.)

7 Likes

Folks in this thread may be interested in this: Releasing the source of the BigMap PoC demo

3 Likes

I was hoping that canisters would simply have access to more memory through orthogonal persistence. Is there going to be some API that will be required? I think to be of the greatest benefit to developers, the heap of the canister needs to be greatly increased. You should be able to just create data structures and fill them with data, scaling up to the capacity of the subnet.

1 Like

That would require the wasm64 WebAssembly extension, which, as Akhi says further up, isnā€™t ready for use yet. And even if it was ā€“ Orthogonal Persistence and keeping all data in the heap, while fine for immutable canisters, doesnā€™t really work well when you want to upgrade your canister. At least not with the programming languages we have at hand.

1 Like

We need to figure this out (upgrading canisters with large Wasm heaps). I think the most exciting long-term goal is an unbounded Wasm heap. Thatā€™s the most ideal solution for developers.

5 Likes

I donā€™t want to imply anything but you just copied @lastmjs comment on this thread

And you did the same for another 11 posts on this forum.
Are you a bot?

7 Likes

Great catch @coin_master

1 Like

Interesting :rofl: lol

I thought folks here would appreciate this update: Two questions about canister storage - #18 by akhilesh.singhania

6 Likes

any news on this? can motoko delveopers make use of the increased stable storage yet?

1 Like

We actually have had support for 32 bit stable storage silently avialable for a while as library ExperimentalStableMemory.mo. We didnā€™t advertise this to avoid people taking a dependency on it.

Iā€™m currently revamping this to support 64-bit stable memory (about half way there). However, weā€™re not convinced this is a great solution for users since it essentially introduces one big global variable (a growable byte array) and has the potential to be a big footgun if used improperly, as well as getting in the way of future, more abstract uses of stable memory including maintaining stable variables in IC stable memory throughout computation, rather than copying out and in on upgrade.

6 Likes

Just to clarify, does that mean that Motoko stable variables are still currently limited to holding only 4 GB worth of data? (Due to stable memory addresses still being 32-bit, i.e. 2^32 ~ 4 GB?)

Yes, and thatā€™s unlikely to change any time soon. It would have rather big implications for our 32-bit tagged value representation.

Gotcha. Hopefully we can unify the APIs for <4 GB stable memory via stable variables and for >4 GB ExperimentalStableStorage at some pointā€¦