I would love access to said document
It seems the code for 64 bit stable memory is live! The IC runs this revision:
~/dfinity/ic $ curl https://ic0.app/api/v2/status|cbor2yaml.rb|grep impl_version % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 100 331 100 331 0 0 565 0 --:--:-- --:--:-- --:--:-- 564 impl_version: 27e1eadbcbe90abfe56d9c8dfd39e1a78e52c624
which, according to the code dump repo history, is dumped as 89446f5
~/dfinity/ic $ git log --oneline 35dd8f9 Update from revision 8a5b9a2e1468dfb286c77084a9b3597b9e3993b5 e362530 Update from revision d8dabe1cb0bb1e60a11e5bb8033d1615fefb252f 89446f5 Update from revision 27e1eadbcbe90abfe56d9c8dfd39e1a78e52c624
and indeed it’s there ic/system_api.rs at 89446f5a04f053040b4863eab5458446d925ed0e · dfinity/ic · GitHub.
So early adopters can probably start playing around with it, even if it didn’t make it to The Internet Computer Interface Specification :: Internet Computer or GitHub - dfinity/ic-hs: A haskell toolbox for the Internet Computer yet.
It would be helpful to the outsiders if the code dumps would include a changelog (maybe auto-generated from the internal repo?), to easier see what changes to look out for.
@nomeata : no the code is not actually live yet. These are stub implementations. See ic/system_state_accessor_direct.rs at 89446f5a04f053040b4863eab5458446d925ed0e · dfinity/ic · GitHub.
Some more of the implementation will get pushed out in the next upgrade. However, we are also planning on doing an NNS proposal where we will ask the community to vote on whether or not this feature should be enabled or not. Only if the vote passes, then we will push the final changes that will enable the feature.
What would be the ETA of getting these system APIs into Motoko once they are live?
cc: @claudio . Maybe he can share his thoughts.
This project is our guinea pig project for what public roadmap projects will look like, and the next 7 days will have a lot of activity surrounding it.
Today, August 25 - Community Conversation on Increased Canister Storage
Thursday, August 26 - Draft proposal posted on the forum for review. We will post to the developer forum thread on the project a markdown doc (hosted on Github repo for NNS proposals) that explains the project design and intent so people can see what the project is about and how we intend to achieve the desired goal.
Wednesday, Sep 1 - This is the nuanced part: I will submit to the NNS a proposal (without a binary that upgrades the IC) asking the wider IC community to vote on whether the Foundation and wider community should continue working, and ultimately deploy, the work for increasing smart contract canister storage. We are essentially using the NNS’s voting mechanisms to let the community express themselves.
Friday, Sep 3 - NNS Proposal has a 48-hour expiration so the decision will be completed by then.
There will be two possible outcomes from this:
a. Proposal passes - if it passes, then the project team will submit an NNS proposal to actually upgrade the IC. In the case of Increased Canister Storage, this will happen relatively quickly since a lot of the groundwork had been done so the actual changes will be light. This is an exception. Many times in the future, if a proposal to “work on this version of the plan” passes, it may take weeks or months to have an implementation ready.
b. Proposal fails - I think this is very unlikely, but worth considering that this means the community does NOT want this change. In which case, we will not continue to work on this plan, but potentially go back to the drawing board and discuss it further with the community
Please note: in order to let the community truly express themselves, the DFINITY Foundation will vote as late as possible. We know lots of people’s neurons follow the DFINITY Foundation so we want to let the community time to breathe.
Our Community Conversation on Increased Canister Storage with @akhilesh.singhania goes live in a few min:
Join the conversation to be able to participate in our live Q&A
Will DFINITY or the ICA be “banned” from voting on this proposal? How can we ensure the vote actually represents the will of the community if very large players vote?
Not sure if you saw this in my note above.
I don’t want to sound more opinionated or “final” than i actually intend so I think I should write the facts and the intents:
- there is no mechanism to ban anyone from voting
- I think this is a very popular proposal
- the foundation intends to get as much community feedback as possible
- the foundation intends to Vote last (with the knowledge of what the community voted), but it does intend to vote (as member of community)
- this is very much an interactive and iterative process so If there are any unintended consequences of this model, we will iterate. Not set in stone. Maybe for more controversial proposals, we can try something different.
As we discussed earlier, we are posting our plan for Increased Canister Storage. You can read our intent, design decisions trade-offs, caveats, and rollout plan.
As Diego mentioned, the current plan is that on Wednesday September 1, Diego will submit an NNS proposal for the community to vote whether the final version of the plan should be implemented.
Please give us feedback, so we can improve and give the IC the best possible plan to increase smart contract canister stable memory.
Currently canisters are effectively limited to 4GiB of storage. This is because stable memory uses the 32-bit addressing scheme and when a canister is upgraded, its Wasm memory is wiped.
A number of applications can benefit from access to additional storage without having to partitioned into multiple canisters. The goal of this proposal is to introduce a stable memory API that allows canisters to address more than 4GiB of memory allowing canisters to have more storage bound [eventually] only by the actual capacity of the subnet.
The goal of this proposal is to increase the amount of memory that canisters can access [eventually] bound only by the actual capacity of the subnet. Since, the Memory64 proposal is not standardized yet and its implementation in Wasmtime is not production ready yet, this proposal enables the increase by introducing a new stable memory API.
When the stable memory was first designed and implemented, the Memory64 proposal was not standardized yet and only recently it was implemented in Wasmtime but the implementation is not production ready yet. Further, the multiple memories proposal is still in Phase 3. With this in mind, stable memory was designed with a 32-bit addressing scheme so that it could eventually be replaced by Wasm native features.
Supporting 64-bit Wasm memory presents a number of performance challenges for the Internet Computer. Currently Wasmtime can efficiently eliminate bounds checks for 32-bit memory accesses by using guard pages. A similar optimization is not possible for 64-bit memory accesses making them more expensive. Besides that more optimization work around
memory.grow() needs to be done before the implementation is production ready.
Since bringing the Memory64 support to IC may take a while and some canisters need large memory now, we propose an extension of the stable memory API to 64-bit instead.
We propose to add four new functions to the System API that mirror the existing 32-bit functions:
ic0.stable64_write: (offset: i64, src: i64, size: i64) -> ()
- Copies the Wasm memory region specified by
sizeto the stable memory starting at the given offset. Note that this API uses 64-bit addressing for the Wasm memory even though at the moment the Wasm memory only supports 32-bit addressing. This is done to keep the possibility of supporting 64-bit Wasm memory open in the future.
- Copies the Wasm memory region specified by
ic0.stable64_read: (dst: i64, offset: i64, size: i64) -> ()
- Copies the stable memory region specified by
sizeto the Wasm memory starting at the given address
- Copies the stable memory region specified by
ic0.stable64_size: () -> (page_count: i64)
- Returns the number of 64KiB pages in the stable memory as a 64-bit integer. Note that it would be possible for this function to continue to return a 32-bit integer as the 32-bit version of the API does. With the page size of 64KiB, a 32-bit integer could address up to 256 TiB which could be sufficient for a very long time. However, it was felt that there should be a clear distinction between the 32-bit and 64-bit versions of the API and having this API return a 64-bit integer should not have any negative impact.
ic0.stable64_grow: (additional_pages: i64) -> (old_page_count: i64)
- Tries to grow the memory by
new_pagesmany pages containing zeroes. If successful, returns the previous size of the memory (in pages). Otherwise, returns
- Tries to grow the memory by
As the specification repo of the Internet Computer is not open source yet, please see the proposed diff here.
In order to ensure smooth transition and upgrade we allow canisters to use the 32-bit and 64-bit versions interchangeably up to 4GiB. In other words, both versions operate on the same stable memory. As soons as the size of the stable memory grows beyond 4GiB the 32-bit versions cease to work. Calling them will result in a trap.
The main risk is canisters mixing the 32-bit and 64-bit functions after the stable memory grows beyond 4GiB. We somewhat mitigate the risk by ensuring that 32-bit functions will trap in such a case so that the canister execution stops instead of continuing with a wrong result.
- One alternative is to introduce a completely new 64-bit stable memory that is disjoint from the existing 32-bit memory. While it is a cleaner design, it would complicate canister upgrades because canisters would need to copy the existing state from the 32-bit stable memory to 64-bit stable memory.
- Another alternative we considered was to say that once a canister uses the 64-bit version of the API then using any of the 32-bit API will result in a trap. We felt that this would allow canisters to more easily detect problems in switching between APIs however we felt that this may overly complicate the implementation.
- Another alternative is to wait until the Memory64 and Multiple Memories proposals are production ready. Then the stable memory can be represented as one of the multiple 64-bit memories removing the need for the new System API.
When introducing a new functionality to any production environment, there are two types of risks that should be managed:
- Regardless of all the testing done before the rollout, there is still a risk of the new functionality introducing some bugs in production.
- Regardless of all the initial feedback gathered about the new API, using it in production may reveal some shortcomings requiring adjustments to the API impacting the canisters that already depend on the API.
Hence, the rollout plan for this proposal would be the following:
- Even though the new API allows canisters to address the entire capacity of the subnet, the stable memory of a given canister will initially be capped at 8GiB.
- The API will be marked as experimental and use of the API in essential canisters will be strongly discouraged. This way the community can feel encouraged to make and accept future proposals to break existing canisters using the API in order to not support deprecated APIs thereby keeping the API as clean and easy to understand as possible.
- After we gain confidence in the API and in its implementation, over subsequent NNS proposals, we will gradually increase the size of the stable memory and mark it as no longer being experimental.
There was a question about how we will test this feature before we deploy it to production. I will write a small text here about how we do testing in general. I will also ask the testing team to write a more detailed post to provide more details.
- As we use Rust, we rely heavily on its unit testing framework for writing unit tests
- Then we have a e2e integration testing framework where we can bring up a simplified version of the Internet Computer on a single machine and validate a number of properties.
- Next we have the reference implementation of the Internet Computer which is used to validate a number of additional properties.
- Next we have a number of testnets where we deploy fully functioning Internet Computers to further test in an environment as close to production as possible.
We have a CI / CD pipeline that runs various tests using the above mechanisms. We run some tests on each PR as it is merged and we run some tests on an hourly basis, some on a nightly basis, etc.
Thanks! All sensible and well done. I endorse this proposal! (I should probably get a neuron that people can follow if they like my endorsements.)
I think an even better argument for returning
i64 here is that the
memory.size Wasm instruction will return an
i64 in the
memory64 extension, and the stable memory API should stay close to the wasm instructions it stands in for.
I like the word “yet” in this sentence.
Your prose sounds as if accessing the first 4GB will trap if the memory is large than 4GB, which I think is good. But the diff adds the check
if offset+size > 2^32 then Trap
This probably should say
if |es.wasm_state.stable_mem| > 2^32 then Trap
I expect soon there will be a demand for a way for canisters to find out how much memory they can use (besides trying). But I don’t mind that not being part of the API, as there is no such functionalities for real WebAssembly memories, and sticking close to the native Wasm experience is probably useful.
Thank you for providing this information. I’d really like to learn more about the testnet environment. Hopefully there will be a lot of detail on this topic in the test team’s post.
Foundation actually has a dedicated testing team so we have asked them to help document and write more about this for folks. thanks for the feedback!
pack it up, we are good to ship then
You totally should!
Oh, and more more issue with the proposed spec: the code for
stable_grow must return -1 if the new size would be above 2^32 bytes. The current code would allow a 32 bit API using canister to grow the canister above 4GB (and then start trapping in every call).
In all seriousness, thank you for taking the time to read in earnest, Joachim
Thanks @akhilesh.singhania for your detailed technical explanation of this change. Thanks @diegop for your follow-through and involving different teams. I do think that proposal adds significant value to the development of IC; with the increased canister storage.
I agree, in part, with @lastmjs on the voting issue with Dfinity Foundation. i.e. I believe that Dfinity Foundation should abstain from voting (even if the proposal fails); even if there is no mechanism to ban Dfinity Foundation from voting.
Democracy, even liquid democracy, assumes (nay requires) that all voting participants are equally informed about all issues surrounding the topic that they are interested in voting on. In software development, it is NOT sufficient to just to show a diff to the voters who do not have access to the source code versus others (in Dfinity Foundation and others bound by NDAs) who do. In asking for such a vote, we are inadvertently creating different classes of voters; one class that is not completely informed (unless the source code is open sourced) and one class that is informed through access to source code. By not providing access to Unit and Integration Test cases, we are once again creating a divide. We are saying to the folks that do not have access, please trust the ones that do have access in their testing.
I disagree, in part, with @lastmjs in context of how bigger players have bigger say in the voting. This is the underlying structure that I believe that I agreed to when I decided to participate in the IC eco-system. I understood my neuron may not have the same voting power as someone else’s neuron. I am ok with it.
The notion of abstinence in voting should certainly not alien; especially since the source code is expected to be open-sourced at some time.