Introducing WASI for IC

check out this interesting :

https://internetcomputer.org/blog/news-and-updates/team-spotlight-node

2 Likes

@lastmjs ,

I am digging into the use of wasi2ic within both Kybra and Azle.

I noticed that both are using the feature transient for the ic-wasi-polyfill, with reference to this issue:

What exactly is the impact of using this feature, and do you know if there is work on-going to fix it?

If you turn that off the polyfill will use the stable memory file system. We tried it and unfortunately found it to be unstable (ha) in various ways that did not make us feel comfortable using it yet.

@sgaflv can give you more details on timelines and current state of the polyfill and its stability.

2 Likes

@lastmjs I am currently working on improving the work of stable-fs. I will provide a new version by the end of this month and will check that the issues are resolved.

3 Likes

@sgaflv , @lastmjs

Just a quick note that I was able to integrate the polyfill and wasi2ic into icpp-pro, the CDK for C++.

It is all still in my dev environment, and I need to do a lot more testing, but first experiments show that it is working really well also in the C++ world, and I believe this is going to help a lot with porting C++ libraries to the IC.

I took a lot of inspiration from kybra, and icpp-pro now installs rust + rust dependencies in a .icpp subfolder of the user’s home directory.

When a user builds their wasm, everything is transparently included. They don’t need to do anything to their code.

Some things that work now (tested it):

  • std::cout : Prints to dfx console
  • file open/close/write/read
  • …
1 Like

@sgaflv ,

Will stable-fs also work for a C++ canister?

If so, do you happen to have a simple demo example available? :pray:*(please?)*

The demo2 that you created was extremely helpful in figuring out how to get it all to work.

1 Like

yes, it should work with the file system as well. Sounds like a good idea to create an extra demo for that (or maybe make the demo2 more advanced). I will add an issue for that.

2 Likes

I am trying out an idea where the user keeps a local copy of sqlite in sync with the main copy running inside the canister. It might be an interesting design pattern for personal apps where the db is known to be small.

All works well, mostly. I am not able to upgrade the canister. The DB is still there after the upgrade but seems to have lost one of its tables.

#[init]
fn init() {
    ic_wasi_polyfill::init(&[0u8; 32], &[]);

    DB.with_borrow_mut(|db| {
        *db = Some(Connection::open(DB_FILE).unwrap());
        let db = db.as_mut().unwrap();

        modules::change::init_db(db);
        modules::chat::init_db(db);
    });
}

#[post_upgrade]
fn post_upgrade() {
    ic_wasi_polyfill::init(&[0u8; 32], &[]);

    DB.with_borrow_mut(|db| {
        *db = Some(Connection::open(DB_FILE).unwrap());
    });
}

Am I missing something here? Or maybe issues are related to the unstableness described by @lastmjs above?

Frontend: https://vicki-naaaa-aaaal-qjgma-cai.icp0.io
Backend: DFINITY Canister Candid UI

2 Likes

@kristofer ,
I wonder if your issue is related to something @ulan fixed and described here

Thanks for the tip. Yes, assigning the wasi memory a MemoryId sounds like good practice. I added that, problem remains though.

More details on the sqlite error code below. Seems sqlite can’t access the db file after upgrade. I have verified using std:fs that the file still exists and file size remains unchanged etc.

(1802) SQLITE_IOERR_FSTAT
The SQLITE_IOERR_FSTAT error code is an extended error code for 
SQLITE_IOERR indicating an I/O error in the VFS layer while trying to 
invoke fstat() (or the equivalent) on a file in order to determine 
information such as the file size or access permissions.

Hello Kristofer,

I’ve checked your example. Unfortunately there seems for be a bug in the way the stable-fs is storing the data in stable structures, which leads to loosing files after canister upgrade. I am working on a fix now.

You have an awesome sqlite example! If you don’t mind, I’d mention it on the wasi2ic page.

5 Likes

Thanks for looking into this! And yes, please do mention my example if you will. :star_struck:

Perhaps canister upgrade could be added to your demo3 as well?

@kristofer
Good point, I’ll update demo3. Check out the latest published ic-wasi-polyfill version 0.4.3, see if it resolves your issue.

3 Likes

Yes, great! Issue resolved!

2 Likes

@lastmjs: fyi, this should also fix the flaky issue you were observing. ^

3 Likes

@sgaflv ,

FYI, the integration of ic-wasi-polyfill & wasi2ic into icpp-pro is finished, and released (as a pre-release initially, but it is working great so far)

You can check out the wonderfully simple C++ code, with stdio, file-io, file upload/download that this now enables in the icpp-demos/canisters/files example canister.

Also fyi, with extreme little effort I was now able to get files to upload to a C++ LLM that I am working on.

I can’t thank you enough for building wasi2ic. :grinning: :pray:

4 Likes

Yes it’s amazing! I really hope it can all get integrated into dfx eventually so that Wasi becomes the defacto standard on ICP and it just works.

3 Likes

@sgaflv ,
from C++, we call raw_init, like this:

int main() {
  raw_init(nullptr, 0);
  return 0;
}

Is that still correct?

I am concerned about the discussion above regarding Rust based aplications, that one should call init_with_memory or init_with_memory_manager.

Not sure if this applies to C++ as well.

@icpp it is only important if your C++ application also wants to directly access the stable memory. The Rust memory manager allows to share the stable memory between the application and the ic-wasi-polyfill library such that they don’t overwrite each other’s data in the stable memory.

If your C++ application stores data in the stable memory using the standard file system and doesn’t call the stable memory API functions, then it is all good and will work correctly because ic-wasi-polyfill will manage the stable memory.

1 Like

I’m really fascinated by the technology behind this!. Could you explain how WASI fits into the ICP ecosystem? I’m curious about what this enables and how it is better than other WASI usages.