Ledger Wasm Too Big

[domwoe]

@PeanutNJam

What I described above is for Musall, where I used the dfinity/examples/motoko/defi/ ledger versions repo at: examples/motoko/defi at master · dfinity/examples · GitHub (its up to date)

This version of the ledger seems too big. It will only work if you install it on a system subnet as described earlier in this thread.

[Severin]

@PeanutNJam

The funny thing about the optimiser is that I build it with cargo install and gives a message of successful, yet when I cargo build I get this → ignoring invalid dependency ic-cdk-optimizer which is missing a lib target.

It’s not intended to be in the dependencies. dfx automatically calls ic-cdk-optimizer to reduce wasm size AFTER compilation if it’s installed.

[PeanutNJam]

Thank you both for assisting me. I got it downloaded, and its unzipped and seems to run at least a bit before an internal error stops the canister build. At least I have overcome my initial error with a too large wasm, and learned some things, now I get a different error → progress I guess. FYI I put the error here as its internal in .rs and you might know where this is happening →

Installing code for canister fileupload, with canister ID rrkah-fqaaa-aaaaa-aaaaq-cai
Installing code for canister ledger, with canister ID ryjl3-tyaaa-aaaaa-aaaba-cai
[Canister ryjl3-tyaaa-aaaaa-aaaba-cai] Panicked at 'Deserialization Failed: "Cannot parse header "', /ic/rs/rust_canisters/dfn_core/src/endpoint.rs:50:41
Error: Failed to install wasm module to canister 'ledger'.
Caused by: Failed to install wasm module to canister 'ledger'.
  Failed to install wasm in canister 'ryjl3-tyaaa-aaaaa-aaaba-cai'.
    Failed to install wasm.
      The Replica returned an error: code 5, message: "Canister ryjl3-tyaaa-aaaaa-aaaba-cai trapped explicitly: Panicked at 'Deserialization Failed: "Cannot parse header "', /ic/rs/rust_canisters/dfn_core/src/endpoint.rs:50:41"

This happens regardless of the new dfx.json remote settings being present in the .json as follows - >

      "remote": {
        "candid": "src/ledger/ledger.public.did",
        "id": {
          "ic": "ryjl3-tyaaa-aaaaa-aaaba-cai"
        }
      }

I will keep hacking at it and see how far I can get with this new error.

[PeanutNJam]

thanks for clarifying, I was silly, and thought that since its on crate.io, it should be added to dependancies.

[lastmjs]

@Severin

dfx automatically calls ic-cdk-optimizer to reduce wasm size AFTER compilation if it’s installed.

I can’t seem to get dfx to do this for me automatically, I’m using dfx 0.11.0. Is it supposed to just happen during the dfx deploy command?

[Severin]

@lastmjs

I’m using dfx 0.11.0. Is it supposed to just happen during the dfx deploy command?

We are bad at communicating, that’s all. This entry in the release notes means that ic-cdk-optimizer is no longer needed. The same optimization is now done through ic-wasm. And ic-wasm is included in dfx now, so you don’t need an external installation of ic-cdk-optimizer anymore.

This only applies to canisters with "type": "rust", so custom canisters still would need manual calls to some optimizer.

Let me see if I can get that line in the changelog clarified. (And maybe even our communication improved?) EDIT: does this help? PR here and here

[lastmjs]

Thanks! So I’m using type: custom canisters (building Azle, the TypeScript CDK). I’d like to remove the need for the user to install ic-cdk-optimizer and have it run for them on dfx deploy. Even if I have to run it for them, I’d love to use the new library that doesn’t require installing ic-cdk-optimizer. I’ll look at those links and hopefully I can figure it out.

[Severin]

@lastmjs

Even if I have to run it for them, I’d love to use the new library that doesn’t require installing ic-cdk-optimizer.

I doubt this is possible with dfx as-is. It doesn’t expose anything that will only optimize the wasm.

[lastmjs]

I’d love to be able to turn this feature on somehow for canisters of type custom, perhaps the dfx.json could have a configuration for this.

[paulyoung]

I’m AFK but I’m curious if ic-wasm gets put in the cache directory by dfx (like moc, etc)

Either way, it looks simple enough to call manually:

GitHub - dfinity/ic-wasm: A collection of libraries and tools for...

@lastmjs not sure if this lends itself well to your use case though.

[paulyoung]

@Severin, in Post-build step for asset canister fails because directory is considered to be outside the workspace root · Issue #2199 · dfinity/sdk · GitHub you were open to examples for reworking the custom build command.

I’m not sure if this topic qualifies but it at least seems related.

[Severin]

@paulyoung

I’m curious if ic-wasm gets put in the cache directory by dfx (like moc, etc)

No, we use ic-wasm as a crate in dfx because bundling an executable in the cache is more annoying. (Code here if you’re interested)

In case you want to look at the cache you can do so with ls "$(dfx cache show)"

@paulyoung

In Post-build step for asset canister fails because directory is considered to be outside the workspace root · Issue #2199 · dfinity/sdk · GitHub you were open to examples for reworking the custom build command.

I’m not sure if this topic qualifies but it at least seems related.

I think having a way to run ic-wasm on the produced wasm is a very reasonable ask since this problem will not only happen with Rust but other languages as well. Added to our backlog.

[paulyoung]

I just started using ic-wasm and it cut the file size in half.

[lastmjs]

How are you using ic-wasm? As in during the build process, how are you setting that up?

[paulyoung]

I’m using nix flakes, so I have this entry in my flake.nix file which sets up the binary:

          packages.ic-wasm = naersk-lib.buildPackage rec {
            pname = "ic-wasm";
            root = pkgs.stdenv.mkDerivation {
              name = "ic-wasm-src";
              src = pkgs.fetchFromGitHub {
                owner = "dfinity";
                repo = "ic-wasm";
                rev = "2e876e84953e24e6a1820aa524f228c8edea4307";
                sha256 = "sha256-0E7Qa0tOtFwV6pkZsjvkGE2TGaj/30+JSlNGtiU0xYo=";
              };
              installPhase = ''
                cp -R --preserve=mode,timestamps . $out
              '';
            };
            cargoBuildOptions = x: x ++ [
              "--package" pname
            ];
            cargoTestOptions = x: x ++ [
              "--package" pname
            ];
          };

Then I call ic-wasm as a post-install step when building my backend canister:

          packages.backend = naersk-lib.buildPackage rec {
            pname = "my-project";
            root = ./backend;
            nativeBuildInputs = [] ++ pkgs.lib.optionals pkgs.stdenv.isDarwin [
              pkgs.libiconv
            ];
            cargoBuildOptions = x: x ++ [
              "--package" pname
              "--target" "wasm32-unknown-unknown"
            ];
            cargoTestOptions = x: x ++ [
              "--package" pname
              "--target" "wasm32-unknown-unknown"
            ];
            compressTarget = false;
            copyBins = false;
            copyTarget = true;
            postInstall = ''
              MY_PROJECT_WASM=$out/target/wasm32-unknown-unknown/release/my_project.wasm
              ${packages.ic-wasm}/bin/ic-wasm -o $MY_PROJECT_WASM $MY_PROJECT_WASM shrink
            '';
          };

[bitdivine]

Actually, I am curious why adding ciborium reduced the wasm size. serde_cbor is not removed from the lockfile in that PR. Does serde_cbor generate a mass of code using macros? At the moment it looks like a case of: “I made it bigger so that it would become smaller”, which usually only works when the original is about to implode anyway.

[roman-kashitsyn]

serde_cbor is not removed from the lockfile in that PR.

We had to continue using serde_cbor for transaction deduplication reasons. Luckily, there we needed only the encoding logic.

Does serde_cbor generate a mass of code using macros?

The bloat seems to come from serde_cbor’s decoding logic (most likely, from generic function instantiations), which we replaced with ciborium in the referenced commit.

[kentosugama]

As an update on the optimizers, ic-wasm now wraps wasm-opt similarly to how ic-cdk-optimizer use to work.

Canister Optimizer available in dfx 0.14.0

[image] Wasm-opt: A general purpose WASM optimizer available in dfx now I’m excited to announce the integration of wasm-opt, a general purpose wasm optimizer, into dfx. You can enable optimizations in your dfx.json like so: { "canisters": { "my_canister": { "optimize": "cycles" } } } You can check out the specific numbers here: Enable wasm optimizer from `dfx 0.14.0` by kentosugama · Pull Request #55 · dfinity/canister-profiling · GitHub As a rough estimate, you can expec…