Does this only affect Rust implementation or is it going to be available for Motoko as well?
Why is it important for you that multi-memory and memory64 are being standardized?
- If it was ever removed from being on track to standardisation, then it would be rather unlikely that Wasmtime would keep it in, so the optimisation @abk suggests would no longer work.
- If it was a proper Wasm feature, then stable memory would be even more efficient (and more flexible) than under this optimisation.
Does this only affect Rust implementation or is it going to be available for Motoko as well?
This is an optimisation in the replica runtime, so would apply equally regardless of implementation language of a canister.
I was under the impression that DFINITY maintains a fork of wasmtime to expose their own host API anyways? Couldn’t they just keep those features then, even if wasmtime decided to remove them?
I think there’s a couple of paths we could take there. One would be to reserve memory index 1 for the stable memory which we inject during instrumentation and if the canister directly uses other memories shift the indices up by one. This is what we already do when injecting additional functions and imports.
Another option could be to have canisters choose between using multiple memories and the stable memory system API. That is, if the canister declares more than one memory we treat the second one as the stable memory and disallow any imports of ic0.stable*
functions.
What do you think about those ideas?
Actually parity-wasm
was marked as unmaintained and deprecated 2 weeks ago. So we’ll need to replace it with something else regardless of if this proposal goes through.
@cryptoschindler: Dfinity doesn’t maintain a fork of Wasmtime - we use the official releases put out by the Bytecode Alliance. The system API is imlemented using the Wasmtime Linker
which let’s the host expose functions that the Wasm module can then import.
@ohsalmeron: @rossberg is correct that this would apply to Rust, Motoko, and any other language used to write canisters.
Why is that necessary? Can’t you inject new definitions at the high end of the index space? Then no shifting is needed. And that should work just fine for stable memory as well.
There should be a relevant design doc from a few years ago, but I can’t find the doc now – I believe it was called “Refining Persistence”. That sketched a design for Wasm-native stable memories where a module can select different stability levels by importing memory from the system using a specified import naming scheme that denotes the desired choice of semantics.
Good point, injecting stable memory at the end would be easier. We have to do shifting when injecting an imported function because the import section comes before the function section which means imported functions all get indices lower than module-defined functions. Is there some trick to get around that? We could then remove the shifting logic
I found it. Yeah, having the import name specify the semantics is a cool idea.
I’m not sure about the exact timeline now, but it would be something on the order of months.
Ah, yes, I see. No, I’m afraid that can’t be avoided.
It’s really exciting to see this finally proposed.
It would also be nice to get a refreshed API for accessing stable memory, perhaps one that doesn’t require making explicit system API calls. Would it be technically possible for canister code to just declare a variable as “stable” (kind of like in Motoko), and updates to that variable’s state get transparently converted into the appropriate system API calls?
Are you saying you’d like something like this in Rust? I guess that would need to be done by a library anyway - even if we give canisters direct access to the multiple memories, the Rust standard library is always going to allocate in memory 0
.
Note that the ic_stable_structures
crate already provides this for the case of BTreeMap
s. Maybe it wouldn’t be too much work to add something like your asking for types that implement the Storable
trait. Any opinion on that @ielashi ?
The walrus
crate avoids the index shifting by using an id_arena for function ids. It basically renumbers everything when emitting the wasm module from AST. From the user/replica respective, it’s totally transparent.
ic-stable-memory
also provides a notion of stable variables in rust.
Yeah, we’re aware of the walrus
crate, but unfortunately it’s not being maintained anymore. We’ve spoken with the authors of it and it doesn’t sound like they have any plans to pick it up again.
It’s too bad because I really like that the added structure in walrus
prevents an entire class of bugs that could appear during instrumentation.
I think some people were seeing this in general yesterday (e.g. https://twitter.com/icircle_nft/status/1691642951984877908?t=R83MBNiT2Xa-MduO3_5XIA&s=19), but I believe it’s been fixed now.