Interesting idea to let canisters directly access stable memory during runtime outside of upgrades.
If they can do that, then what’s the need for a wasm heap? Can’t canisters use stable memory for all computation in that case, including scratch space? Is it slower to access the wasm heap versus stable memory via this hypothetical API? Does it cost more to store data in stable memory?
Thanks!
That was the vision all along: that wasm code has multiple memories they can use, with different lifetimes, and all accessed via Wasm memory instructions, so fast.
The stable memory API through expensive bulk system API function calls was just a make shift. Unfortunately the necessary webassembly extension (multiple memories) didn’t make it in time into the Wasm tools we use, so we are stuck with this transitional API for now. But in principle you are absolutely right.
Hm, when you mean “expensive bulk system API function calls”, do you mean that storing data in stable memory is more expensive than in non-stable memory? (If not because of any differences in storage cycle cost but instead due to the additional computation required to transfer data in and out of stable memory in pre- and post-upgrade functions?)
In terms of actual running time, normal memory is accessed using Wasm memory instructions which are compiled to memory instructions, so that’s native speed fast. Accessing stable memory with the current API requires function calls into the host, bounds checking, explicit copying etc., so definitely more expensive. Whether that cost is currently reflected also in cycle costs (which you probably care more about than running time) I don’t know, but we can assume that cycle costs will be refined to match real costs eventually.
In the original vision, building on Wasm multi memory support, stable memory would be accessed the same as the main memory, and would likely have the same access cost. One wouldn’t speak of “main” memory at that point any more, as there can be any number of memories, of varying lifetimes (per message, maybe per call, per module version, permanent).
It might help if the people at DFINITY could share the “Refining persistence” document by Andreas Rossberg, which explains this better.
I would love access to said document
It seems the code for 64 bit stable memory is live! The IC runs this revision:
~/dfinity/ic $ curl https://ic0.app/api/v2/status|cbor2yaml.rb|grep impl_version
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
100 331 100 331 0 0 565 0 --:--:-- --:--:-- --:--:-- 564
impl_version: 27e1eadbcbe90abfe56d9c8dfd39e1a78e52c624
which, according to the code dump repo history, is dumped as 89446f5
~/dfinity/ic $ git log --oneline
35dd8f9 Update from revision 8a5b9a2e1468dfb286c77084a9b3597b9e3993b5
e362530 Update from revision d8dabe1cb0bb1e60a11e5bb8033d1615fefb252f
89446f5 Update from revision 27e1eadbcbe90abfe56d9c8dfd39e1a78e52c624
and indeed it’s there ic/system_api.rs at 89446f5a04f053040b4863eab5458446d925ed0e · dfinity/ic · GitHub.
So early adopters can probably start playing around with it, even if it didn’t make it to The Internet Computer Interface Specification :: Internet Computer or GitHub - dfinity/ic-hs: A haskell toolbox for the Internet Computer yet.
It would be helpful to the outsiders if the code dumps would include a changelog (maybe auto-generated from the internal repo?), to easier see what changes to look out for.
@nomeata : no the code is not actually live yet. These are stub implementations. See ic/system_state_accessor_direct.rs at 89446f5a04f053040b4863eab5458446d925ed0e · dfinity/ic · GitHub.
Some more of the implementation will get pushed out in the next upgrade. However, we are also planning on doing an NNS proposal where we will ask the community to vote on whether or not this feature should be enabled or not. Only if the vote passes, then we will push the final changes that will enable the feature.
What would be the ETA of getting these system APIs into Motoko once they are live?
Hey Folks!
This project is our guinea pig project for what public roadmap projects will look like, and the next 7 days will have a lot of activity surrounding it.
Today, August 25 - Community Conversation on Increased Canister Storage
Thursday, August 26 - Draft proposal posted on the forum for review. We will post to the developer forum thread on the project a markdown doc (hosted on Github repo for NNS proposals) that explains the project design and intent so people can see what the project is about and how we intend to achieve the desired goal.
Wednesday, Sep 1 - This is the nuanced part: I will submit to the NNS a proposal (without a binary that upgrades the IC) asking the wider IC community to vote on whether the Foundation and wider community should continue working, and ultimately deploy, the work for increasing smart contract canister storage. We are essentially using the NNS’s voting mechanisms to let the community express themselves.
Friday, Sep 3 - NNS Proposal has a 48-hour expiration so the decision will be completed by then.
There will be two possible outcomes from this:
a. Proposal passes - if it passes, then the project team will submit an NNS proposal to actually upgrade the IC. In the case of Increased Canister Storage, this will happen relatively quickly since a lot of the groundwork had been done so the actual changes will be light. This is an exception. Many times in the future, if a proposal to “work on this version of the plan” passes, it may take weeks or months to have an implementation ready.
b. Proposal fails - I think this is very unlikely, but worth considering that this means the community does NOT want this change. In which case, we will not continue to work on this plan, but potentially go back to the drawing board and discuss it further with the community
Please note: in order to let the community truly express themselves, the DFINITY Foundation will vote as late as possible. We know lots of people’s neurons follow the DFINITY Foundation so we want to let the community time to breathe.
Our Community Conversation on Increased Canister Storage with @akhilesh.singhania goes live in a few min:
Join the conversation to be able to participate in our live Q&A 
Will DFINITY or the ICA be “banned” from voting on this proposal? How can we ensure the vote actually represents the will of the community if very large players vote?
Not sure if you saw this in my note above.
I don’t want to sound more opinionated or “final” than i actually intend so I think I should write the facts and the intents:
The facts
The Intent
As we discussed earlier, we are posting our plan for Increased Canister Storage. You can read our intent, design decisions trade-offs, caveats, and rollout plan.
As Diego mentioned, the current plan is that on Wednesday September 1, Diego will submit an NNS proposal for the community to vote whether the final version of the plan should be implemented.
Please give us feedback, so we can improve and give the IC the best possible plan to increase smart contract canister stable memory.
Main authors: @ulan , @akhilesh.singhania
Currently canisters are effectively limited to 4GiB of storage. This is because stable memory uses the 32-bit addressing scheme and when a canister is upgraded, its Wasm memory is wiped.
A number of applications can benefit from access to additional storage without having to partitioned into multiple canisters. The goal of this proposal is to introduce a stable memory API that allows canisters to address more than 4GiB of memory allowing canisters to have more storage bound [eventually] only by the actual capacity of the subnet.
The goal of this proposal is to increase the amount of memory that canisters can access [eventually] bound only by the actual capacity of the subnet. Since, the Memory64 proposal is not standardized yet and its implementation in Wasmtime is not production ready yet, this proposal enables the increase by introducing a new stable memory API.
When the stable memory was first designed and implemented, the Memory64 proposal was not standardized yet and only recently it was implemented in Wasmtime but the implementation is not production ready yet. Further, the multiple memories proposal is still in Phase 3. With this in mind, stable memory was designed with a 32-bit addressing scheme so that it could eventually be replaced by Wasm native features.
Supporting 64-bit Wasm memory presents a number of performance challenges for the Internet Computer. Currently Wasmtime can efficiently eliminate bounds checks for 32-bit memory accesses by using guard pages. A similar optimization is not possible for 64-bit memory accesses making them more expensive. Besides that more optimization work around memory.grow() needs to be done before the implementation is production ready.
Since bringing the Memory64 support to IC may take a while and some canisters need large memory now, we propose an extension of the stable memory API to 64-bit instead.
We propose to add four new functions to the System API that mirror the existing 32-bit functions:
ic0.stable64_write: (offset: i64, src: i64, size: i64) -> ()
src and size to the stable memory starting at the given offset. Note that this API uses 64-bit addressing for the Wasm memory even though at the moment the Wasm memory only supports 32-bit addressing. This is done to keep the possibility of supporting 64-bit Wasm memory open in the future.ic0.stable64_read: (dst: i64, offset: i64, size: i64) -> ()
offset and size to the Wasm memory starting at the given address dst.ic0.stable64_size: () -> (page_count: i64)
ic0.stable64_grow: (additional_pages: i64) -> (old_page_count: i64)
new_pages many pages containing zeroes. If successful, returns the previous size of the memory (in pages). Otherwise, returns -1.As the specification repo of the Internet Computer is not open source yet, please see the proposed diff here.
In order to ensure smooth transition and upgrade we allow canisters to use the 32-bit and 64-bit versions interchangeably up to 4GiB. In other words, both versions operate on the same stable memory. As soons as the size of the stable memory grows beyond 4GiB the 32-bit versions cease to work. Calling them will result in a trap.
The main risk is canisters mixing the 32-bit and 64-bit functions after the stable memory grows beyond 4GiB. We somewhat mitigate the risk by ensuring that 32-bit functions will trap in such a case so that the canister execution stops instead of continuing with a wrong result.
When introducing a new functionality to any production environment, there are two types of risks that should be managed:
Hence, the rollout plan for this proposal would be the following:
There was a question about how we will test this feature before we deploy it to production. I will write a small text here about how we do testing in general. I will also ask the testing team to write a more detailed post to provide more details.
We have a CI / CD pipeline that runs various tests using the above mechanisms. We run some tests on each PR as it is merged and we run some tests on an hourly basis, some on a nightly basis, etc.
Thanks! All sensible and well done. I endorse this proposal! (I should probably get a neuron that people can follow if they like my endorsements.)
I think an even better argument for returning i64 here is that the memory.size Wasm instruction will return an i64 in the memory64 extension, and the stable memory API should stay close to the wasm instructions it stands in for.
I like the word “yet” in this sentence.
Your prose sounds as if accessing the first 4GB will trap if the memory is large than 4GB, which I think is good. But the diff adds the check
if offset+size > 2^32 then Trap
This probably should say
if |es.wasm_state.stable_mem| > 2^32 then Trap
in stable_read, stable_write and stable_size.
I expect soon there will be a demand for a way for canisters to find out how much memory they can use (besides trying). But I don’t mind that not being part of the API, as there is no such functionalities for real WebAssembly memories, and sticking close to the native Wasm experience is probably useful.
Thank you for providing this information. I’d really like to learn more about the testnet environment. Hopefully there will be a lot of detail on this topic in the test team’s post.
Foundation actually has a dedicated testing team so we have asked them to help document and write more about this for folks. thanks for the feedback!
pack it up, we are good to ship then 
You totally should!