Automatically generate candid from rust sources

It’s not about adding a few lines, it’s about the complexity of installing yet another dependency, a Wasm runtime, on an arbitrary developer’s machine. DFINITY has done a great job of getting dfx to work very well across architectures, we currently don’t have the infrastructure setup to easily precompile dependencies across architectures…maybe we’ve been doing it wrong, but it’s been difficult to deal with these issues.

cargo test caused problems in the past because of long compile times and issues compiling across architectures like I mentioned, then we switched to installing a wasm runtime across architectures which as had its own problems, the latest being that wasmer (and we had to switch away from wasmtime because there were issues with wasi) not working on Mac M1s, and now we’re trying to get the Candid out in post_install using __export_service which is problematic for other reasons.

The developer experience for us and our users will be excellent if dfx can remove the requirement to specify a candid file and if it can automatically generate and obtain it from Rust canisters, even if those canisters are declared as custom.

There are at least two problems with retrieving the Candid in post_install:

  1. There is no way to put the Candid into the metadata before the first deploy
  2. init/post_upgrade params don’t work when deploying with dfx because apparently the --argument is not encoded or recognized by dfx since the Candid file is empty (in our case it is empty because on the first deploy we create a Candid file with an empty service, then on that post_install we request the Candid and write it to disk, then on subsequent deploys everything is good unless the init/post_upgrade params change)

I’m currently stuck on this problem right now, 1 isn’t too bad for the moment because __get_candid_interface_tmp_hack is supported by tooling, but 2 seems like a blocking issue (that I just discovered).

Yes this helps a lot, but allowing custom canisters to turn on this option seems the more ideal solution so that we can remove this code entirely from our build process.

It’s not possible to remove this code entirely from the build process. dfx has no knowledge of how your build script works, so you will have to adapt the build script. dfx can certainly help by providing the necessary dependencies, e.g., wasmtime and ic-wasm in the cache. Your current approach for using post_install is certainly problematic, and that’s what we want to fix.

Actually this might not help, as I mentioned above I switched from wasmtime to wasmer because of issues with wasi, but perhaps you’ve overcome those issues in the cdk.

Maybe. We will have to see. Can you show me your build script, so that I can take a deeper look?

1 Like

If dfx sees a custom canister that has a property turned on to automatically generate the Candid from Rust code, couldn’t it look at the wasm binary path in the dfx.json and generate the Candid? I don’t see why this would require dfx to know anything about the custom build script. In fact I don’t even think dfx needs to care about the language, since it’s just running the Wasm binary through wasmtime. If dfx sees a property in the dfx.json instructing it to run the Wasm binary through wasmtime to retrieve the Candid, can’t it do that without needing to know anything else? This could be on by default for Rust canisters and off by default for custom canisters but with the option to turn it on.

Here’s our script on main for Kybra (doesn’t have the local wasmtime/wasmer execution, we now do everything in a post_install script): kybra/kybra/__main__.py at main · demergent-labs/kybra · GitHub

Here’s where we used to run wasmer: kybra/kybra/candid.py at 3e74103a75faeafe6f5c9e6f433cb34888383332 · demergent-labs/kybra · GitHub

Here’s where we used to run Candid test: kybra/kybra/__main__.py at 0.4.0 · demergent-labs/kybra · GitHub

I might have been mistaken about my reasons for not using wasmtime, I think we may have been using wasmer from the beginning, I’m pretty sure I tried wasmtime first and then switched over to wasmer for reasons I don’t now remember.

P.S. we do something similar in Azle, the situation is much better in Azle because Node.js uses V8 which has Wasm support by default so we can just use that to generate the Candid relatively easily: https://github.com/demergent-labs/azle/blob/main/src/compiler/generate_candid_file.ts

So I want to reply to this comment: Ic-cdk wasi feature - #8 by chenyan

Any updated thoughts on allowing custom canisters to turn this functionality on?

By definition, dfx won’t interfere with custom canisters. For Azle, you will have to adapt the build script https://github.com/dfinity/cdk-rs/blob/main/examples/build.sh to use the new ic-cdk feature.

If the difficulty is in the extra dependency of wasmtime and ic-wasm, we will provide the binaries in dfx cache.

I didn’t realize that there were two different binaries, I think I understand some of the issues now. In that case I would ask if that situation can ever be addressed, so that there is only ever one binary produced that the candid can be extracted from and can be deployed. We are doing that right now, one compilation produces a binary that can be executed in a Wasm VM to obtain the Candid, and that same binary can be deployed to the IC.

This is how we’re able to produce just one Wasm binary that the Candid can be extracted from and that can be deployed to the IC:

Essentially we do this in the canister:

        candid::export_service!();

        // Heavily inspired by https://stackoverflow.com/a/47676844
        #[no_mangle]
        pub fn get_candid_pointer() -> *mut std::os::raw::c_char {
            let c_string = std::ffi::CString::new(__export_service()).unwrap();

            c_string.into_raw()
        }

And we do this in Node.js during the build step:

import { readFileSync, writeFileSync } from 'fs';

export function generateCandidFile(candidPath: string, wasmFilePath: string) {
    const wasmBuffer = readFileSync(wasmFilePath);

    const wasmModule = new WebAssembly.Module(wasmBuffer);
    const wasmInstance = new WebAssembly.Instance(wasmModule, {
        ic0: {
            accept_message: () => {},
            call_cycles_add: () => {},
            call_cycles_add128: () => {},
            call_data_append: () => {},
            call_new: () => {},
            call_on_cleanup: () => {},
            call_perform: () => {},
            canister_cycle_balance: () => {},
            canister_cycle_balance128: () => {},
            canister_self_copy: () => {},
            canister_self_size: () => {},
            canister_version: () => {},
            certified_data_set: () => {},
            data_certificate_copy: () => {},
            data_certificate_present: () => {},
            data_certificate_size: () => {},
            debug_print: () => {},
            global_timer_set: () => {},
            instruction_counter: () => {},
            is_controller: () => {},
            msg_arg_data_copy: () => {},
            msg_arg_data_size: () => {},
            msg_caller_copy: () => {},
            msg_caller_size: () => {},
            msg_cycles_accept: () => {},
            msg_cycles_accept128: () => {},
            msg_cycles_available: () => {},
            msg_cycles_refunded: () => {},
            msg_cycles_refunded128: () => {},
            msg_method_name_copy: () => {},
            msg_method_name_size: () => {},
            msg_reject_code: () => {},
            msg_reject_msg_copy: () => {},
            msg_reject_msg_size: () => {},
            msg_reject: () => {},
            msg_reply_data_append: () => {},
            msg_reply: () => {},
            performance_counter: () => {},
            stable_grow: () => {},
            stable_read: () => {},
            stable_size: () => {},
            stable_write: () => {},
            stable64_grow: () => {},
            stable64_read: () => {},
            stable64_size: () => {},
            stable64_write: () => {},
            time: () => {},
            trap: () => {}
        }
    });

    const candidPointer = (wasmInstance.exports as any).get_candid_pointer();

    const memory = new Uint8Array((wasmInstance.exports.memory as any).buffer);

    let candidBytes = [];
    let i = candidPointer;
    while (memory[i] !== 0) {
        candidBytes.push(memory[i]);
        i += 1;
    }

    writeFileSync(candidPath, Buffer.from(candidBytes));
}

We are doing that right now, one compilation produces a binary that can be executed in a Wasm VM to obtain the Candid

IIUC, you are taking a more complicated detour to achieve that. So using the new ic-cdk feature is an improvement.

In that case I would ask if that situation can ever be addressed, so that there is only ever one binary produced that the candid can be extracted from and can be deployed.

Probably not. Not without changing rustc or the rust analyzer. However, developer can also choose an opposite direction similar to protobuf: Developers write candid file as the source of truth to define data types and API. We can then provide tools to generate bindings from candid to the host language. If we go with this approach, we don’t need to generate did file anymore.

Ideally we could remove this code, remove the similar code in Kybra, not be required to generate Candid at all, and remove the need for the developer to specify a path to a Candid file in dfx.json.

Hi @lastmjs - we have a developer tooling working group session tomorrow (8/3) and this will be one of the topics we will discuss. I encourage you to attend to voice what you’d like to see and help us shape the feature.

For the upcoming group session, I summarized my thoughts here.

Unfortunately I wasn’t able to attend the session.

Is the feature to automatically generate .did files from rust code available somewhere? If not, is there an expected timeline for the release? I’m fully with @lastmjs on this. This feature is important for a good developer experience. Spent the last few hours looking for the problem until I realised the automatic generation for rust canisters wasn’t available.

1 Like

Yes you can now generate the did from Rust without workaround but, it needs an additional script / call to wastime afterwards. Ultimately I understand the goal is to make all automatic.

I wrote a post about last improvements and current state, hope that helps:

2 Likes

This is perfect, thanks!

1 Like

Well @peterparker’s solution works up until stable--structures are used.
the workaround for the original workaround involving updating “ic-cdk”
PR
has been rejected.
Is there a way to do it all within ic sdk?

I literally complained about this today and literally also shared the wording “it’s a pain in the a” with my colleagues.

The PR you mention wasn’t a proper solution for a definitive implementation. Team is now trying their best to push the effort.

There is a draft PR opened in the cdk-rs (feat: sdk 1227 by lwshang · Pull Request #424 · dfinity/cdk-rs · GitHub), there will be a new CLI tool and now there are discussing the integration in dfx.

Meanwhile, using above PR (chore: update ic-cdk to 0.10 by lwshang · Pull Request #108 · dfinity/stable-structures · GitHub) remains the workaround I know.

i.e. I generally develop and only when I want to generate the did files, I switch the source of the crate in the toml file.

definitely a pain in the A :sweat_smile: .
seems like I will have to keep both solutions,
ur script and “__get_candid_interface_tmp_hack”

"generate_old": "scripts/build_did.sh && dfx generate backend",
"generate": "cargo test generate_candid && dfx generate backend",

thank you really for the script and pushing this. :pray:

1 Like

Great news @sadernalwis , the stable--structures incompatibility has been resolved by today’s release of the new ic_cdk@0.11.0 and the new CLI tool candid-extractor.

I updated my blog post but, to summarize the difference:

  1. bump ic_cdk and related libraries

  2. install candid-extractor

  3. update the did.sh script I provided as follow

#!/usr/bin/env bash

function generate_did() {
  local canister=$1
  canister_root="src/$canister"

  cargo build --manifest-path="$canister_root/Cargo.toml" \
      --target wasm32-unknown-unknown \
      --release --package "$canister"

  candid-extractor "target/wasm32-unknown-unknown/release/$canister.wasm" > "$canister_root/$canister.did"
}

# The list of canisters of your project
CANISTERS=console,observatory,mission_control,satellite

for canister in $(echo $CANISTERS | sed "s/,/ /g")
do
    generate_did "$canister"
done

Big shout-out to @lwshang for this improvement :pray:. Not all heroes wear capes!!!

2 Likes