Debugging "complexity exceeds the maximum complexity allowed"

Developers
1

Hey, I was testing a Rust canister locally and I got the following error in the dfx replica:

Canister's Wasm module is not valid: Wasm module contains a function at index 0 with complexity 1029495 which exceeds the maximum complexity allowed 1000000.

How can I go about debugging these issues? are there tools which will allow me to inspect the complexity of a function and find the offending one?
Also, what are the recommended ways to reduce a function’s complexity?
I am trying to verify a merkle proof that can be pretty big (proof of membership among >30k elements)

2

wasm-objdump from the wabt package would be my first step. The error says the function is at index 0, so you should be able to see which one it is with wasm-objdump -j Function. Generally this doesn’t happen with a handwritten function, but when you’re using some macros which generate huge amounts of code - is it possible you’re doing something like that? The solution would be to break up this large function into a couple smaller ones.

Edit: my mistake, it should be the code section with wasm-objdump -j Code.

3

This is the truncated output:

Function[6785]:
 - func[30] sig=1 <_ZN100_$LT$serde..de..impls..ArrayVisitor$LT$$u5b$T$u3b$$u20$32$u5d$$GT$$u20$as$u20$serde..de..Visitor$GT$9visit_seq17h528fd31d9258ab95E>
.... more lines below this

Is <serde::de::impls::ArrayVisitor<[T; 32]> as serde::de::Visitor>::visit_seq::h64a6699ae0cae3ae
The offending function?

4

I did a complete objdump instead of just -j Function

Import[30]:
 - func[0] sig=1 <_ZN3ic03ic011debug_print17h2c2d5d086965e01eE> <- ic0.debug_print

Is this the function at index 0? am I printing something that’s too complex?

5

Sorry, the index here actually refers to the code section. We should probably improve the error to clarify that. So try looking at wasm-objdump -x -j Code.

6

Here’s the truncated output

Code[6785]:
 - func[30] size=4942 <_ZN100_$LT$serde..de..impls..ArrayVisitor$LT$$u5b$T$u3b$$u20$32$u5d$$GT$$u20$as$u20$serde..de..Visitor$GT$9visit_seq17h528fd31d9258ab95E>

Which seems to be the same
I checked the implementation of ArrayVisitor and its indeed maco heavy, so I switched to GenericArray<u8, U32>
However I still got the error, albeit with a different complexity:

Failed to install wasm on canister: Principal { len: 10, bytes: [128, 0, 0, 0, 0, 16, 0, 49, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0] }, error: "Error from Canister fvpkz-24aaa-aaaaa-qaazq-cai: Canister's Wasm module is not valid: Wasm module contains a function at index 0 with complexity 1012563 which exceeds the maximum complexity allowed 1000000."

Checking the dump again:

Code[6798]:
 - func[30] size=514 <_ZN10num_bigint6bigint7convert18to_signed_bytes_le17hb08e9bc707f174a9E>

Which is num_bigint::bigint::convert::to_signed_bytes_le::hb08e9bc707f174a9

I am not explicitly calling this function anywhere, and its used by many libraries (ic libs included), so removing its usage isn’t really feasible for me

7

Something seems weird here. I wouldn’t expect either of those functions to hit the limits, especially because their sizes aren’t very big. Maybe if you have some enum type with many variants the serde (de)serialize functions for that type would hit the limit. Would you be able to send me the problematic Wasm?

8

Sure, individual_user_template.wasm - Deupload

9

Did you run optimization on the wasm?

I am asking because I have run into this error when I was trying to optimize a wasm file with binaryen.

10

no i am just using dfx build

1 Like
11

Are you sure the wasm you sent me is the same one that’s giving you trouble? It doesn’t seem to hit the limit when I test it out locally. This is using dfx 0.24.2. I wasn’t able to fully install it because I’m not sure what arguments your canister_init expects, but the Wasm is definitely passing the step where the complexity is checked.

12

Apologies, I was sending the wasm from target/release/wasm32-unknown-unknown instead of
.dfx/! :see_no_evil:

it seems that the error originates from ciborium:

Section Details:

Code[3724]:
 - func[30] size=282794 <_ZN91_$LT$$RF$mut$u20$ciborium..de..Deserializer$LT$R$GT$$u20$as$u20$serde..de..Deserializer$GT$15deserialize_map17h0a05ef429be6690cE>

Here’s the correct wasm for your reference as well:

I’ll look into replacing ciborium

13

I’m not sure if ciborium itself is the problem. The Wasm generated by cargo/rustc is fine (since that’s the first one you sent me and it doesn’t hit this limit), so maybe the dfx post-processing steps are just doing a bit too much inlining or something. Which version of dfx are you using and what’s in your dfx.json file?

In that large function’s implementation I’m seeing calls to some ic_stable_structures::BTreeMap functions, so I think the full decoding of the types that are being stored in this tree might be getting inlined.

14

its dfx 0.24.

Here’s the config for the canister in dfx.json

    "individual_user_template": {
      "candid": "./src/canister/individual_user_template/can.did",
      "declarations": {
        "node_compatibility": true,
        "output": "./export/declarations/individual_user_template"
      },
      "gzip": true,
      "optimize": "size",
      "package": "individual_user_template",
      "type": "rust"
    },

You can also check the source as well(though you can see i’ve switched to minicbor for now to get unblocked): hot-or-not-backend-canister/src/canister/individual_user_template at rupansh/pop · yral-dapp/hot-or-not-backend-canister · GitHub

We’re not doing serialization/deserialization of stable btree map, but the std BTreeMap

15

Yeah that all seems pretty reasonable. It might just be that your CanisterData type is a bit large and that its serialization creates a big function once everything is inlined.

1 Like