Possible BUG: Running into "Payload Too Large" error even though my canister is around 1 MB in size

This is what I’m seeing on my terminal when trying to upgrade a canister:

I can see that an optimized build was created from further up in the log:

My build script looks something like this:

Essentially I’m gzipping the created wasm and dfx directly accepts the *.wasm.gz as the wasm bundle to install.

Here’s what the size of the user_index wasm gzipped looks like:
image

I’m on dfx 0.13.1

I’m kind of stuck on how to proceed. Any inputs are appreciated.

Thoughts?

Tagging @Severin and @lwshang since I had run into this before and had been able to solve this with their help

1 Like

It sounds to me like the install command is not picking up the gzipped file, but the uncompressed wasm instead. How does your dfx.json look like? I think you have to point it to the .gz file

This is what it looks like:
image

I am using the “rust” type instead of “custom”

Also, my config hasn’t changed since I solved this:

I only recently upgraded dfx to v0.13.1 from v0.12

Also, the gzip process removes the bare wasm file. The folder after the gzip step looks like this:

So, not sure where it would pick up the uncompressed WASM from

I see two ways to resolve this. Either you move to canister type custom and point it to the gzipped wasm or you point your build script at the gzipped wasm (using --wasm target/wasm32-unknown-unknown/release/<canister>.wasm.gz).

I haven’t double-checked but I would assume dfx copies the wasm into a local cache inside of .dfx. It does that to do some more transformations like baking the .did file into the right metadata section so it shows up in chain explorers properly

3 Likes

@Severin That fixed it. Thank you so much :slight_smile:

For anyone reading later, the solution that worked for me is to specify the exact *.wasm.gz as an additional argument as part of the dfx canister install .. step

Also, it might make sense to add the gzip step as part of dfx deploy or dfx canister install

3 Likes

We’ve talked a lot about that internally. IIRC there are nondeterminism problems with gzip and that breaks some properties that we don’t want to break

1 Like

Hi @Severin

I believe there’s a bug with the --wasm flag.

My installation script looks like this:

When I install without the --wasm flag, the argument values are properly passed to the canister init function. However, when I use the --wasm flag, these values are not being passed into the canister’s init function. Something I am missing?

Also, I found that when --wasm is specified, Candid UI isn’t installed as well. Is this intended?

Thoughts?

Kinda stuck with this. Appreciate any help you can provide :slight_smile:

Hi @Severin

Let me know if you have any thoughts on this?

If you use --wasm then the argument parsing is indeed different from the one used otherwise (see last arg to the function). But I don’t understand why this would turn out so differently. If parsing fails, it would complain either way

I’m not sure. Reading the code, I get the impression that --wasm is meant for a very manual process, but I’m not sure that it should skip Candid UI entirely. For now I can offer as a workaround that you install any random canister without the --wasm flag because that will then automatically install Candid UI.

@Severin
The Candid UI isn’t a blocker.

But the arguments not being passed is. Without it, we are unable to deploy and test locally.

Any suggestions on how I could help debug this? If you want, I can try creating a minimal reproduction

A reproduction would be perfect. If you want an ugly workaround, you could copy the wasm to the place where dfx would expect it to be and hope that it works. It would expect the wasm to be at .dfx/<network name>/canisters/<canister name>/<canister name>.wasm

1 Like

Hi @Severin

I managed to reproduce this.

Here is a link to the repo:

I am printing the passed arguments in the init function here:

When I am passing the --wasm flag here:

I get this

But when I don’t pass the --wasm, I get both of the arguments as shown here:

Thoughts?

1 Like

Thank you very much for the repro! I get the same result, and don’t like it :smile:

I filed the bug (note to future self: internal ticket) and hope we soon get around to fixing it.

Do you need help to find a workaround for now? I think if you put the gzipped wasm to .dfx/local/canisters/backend/backend.wasm (just strip the .gz extension) then dfx canister installshould pick it up even without the–wasm` flag

2 Likes

Hi @Severin
Thank you for the acknowledgement. I’ll try the manual copying workaround for now.

But we’ll be eagerly awaiting an actual fix as the manual copying would be error prone and would need to be explained to and adopted by frontend for whom the flow was just:

  • pull the latest commit
  • run install script on local replica for local testing

I think you can put it in the install script too and just put a comment there that this weird step is there because of a bug

Let me try and come back if I need more inputs

1 Like

--wasm is used to interface with foreign canisters; it neither requires nor integrates with dfx.json. This means that it does not know about the full candid interface, which interacts badly with a bug in candid involving merging unknown variants into a common type. (The data is actually there, it’s just being serialized with the wrong variant, and HashMap is pruning the duplicated key.) In general you should not be using --wasm as part of a general deployment strategy.

Is this Candid bug reported somewhere? Any way to track progress towards a fix?

Thanks for acknowledging and providing visibility on this. This was a head scratcher for us for a while :sweat_smile:

No idea. Maybe @chenyan would know?

There is an internal ticket on the dfx side to fix this: dfx needs a way to get the type of the init args. Before this gets fixed, you can use ic-repl to deploy the canister. Here is an example: examples/account.test.sh at master · dfinity/examples · GitHub

1 Like