Wasm module defined ___ globals which exceeds the maximum number allowed 200

Developers
21

That is awesome news!!

1 Like
22

@dsarlis ,
I think we can just use this thread to discuss the possibility to completely remove this limit on the number of globals.

The coming increase to 1,000 will be sufficient for all cases I have seen so far, but we’re only getting started, so you never know if this is sufficiently high.

Would it be possible to get rid of it altogether?

1 Like
23

@icpp I think Diego tagged me out of reflex, this is more in the purview of the runtime team nowadays (I’m in the execution team myself which is very closely related and sometimes leading to these confusions). I’ll let the folks of the runtime team answer on the possibility of getting rid of limits. (Andriy who replied that the limit is being increased to 1000 is part of the runtime team).

2 Likes
24

The limits are in place because ATM, on the IC, WASM compilation should happen within a round, i.e., within approximately 1 second.

There are other compilation limits in place as well. So, removing this particular limit for the globals won’t break the IC, but it also won’t help to accommodate much larger programs on the IC, unfortunately.

The fundamental solution is a multi-round compilation, which will be technically possible after the query stats feature is fully implemented.

3 Likes
25

Multi-round compilation ftw!

26

Hello,

Is the change rolled out yet?

27

It should be rolled out to all the app subnets and the latest DFX beta should include the change:

DFX_VERSION=0.15.2-beta.0 sh -ci "$(curl -fsSL https://internetcomputer.org/install.sh)"
3 Likes
28

Sorry to bring this up again. Been building Deep Learning using Flashlight, and its using 1388 globals which exceeds the maximum number allowed 1000. Could we bring this up again to 2500 (in case i need to bring in yet another library)?

2 Likes
29

@ktimam would you mind sharing the build steps on how to reproduce the Wasm binary? I would like to look into why the library uses so many variables and if that is a regular pattern or something exceptional.

30

Thanks @ulan . Can you send me your github id so i can send you an invite to the repo?

31

My github id is ulan

32

Sent you access. Please make sure you’re on latest icpp-pro 3.17.0, then run “icpp build-wasm” from project root, then “dfx deploy”

1 Like
33

Thanks @ktimam! I completed the investigation.

The Wasm binary declares 1387 globals, however only 138 of them are used.

The unused globals are marked as exported as shown below. That is the reason why they are not eliminated by Wasm optimizers.

I think the fix could be implemented on the icpp-pro side:

  • Remove all (export global) declarations by modifying the Wasm binary and run a Wasm optimizer (wasm-opt or ic-wasm). Note that globals don’t need to be exported on the IC. This should remove the unused globals.
  • If that doesn’t help, then it should be possible to implement a script that looks for (global.get X) and (global.set X) statements in the Wasm binary and removes all globals that are not accessed.
  • There might be also a way on the C++ toolchain side to optimize such that globals are not exported in the first place (e.g. using link time optimization (LTO) flags)
  (global (;1386;) i32 (i32.const 1486192))
  (global (;1387;) i32 (i32.const 1486248))
  (export "memory" (memory 0))
  (export "_ZTVN10__cxxabiv120__si_class_type_infoE" (global 139))
  (export "_ZTINSt3__219__shared_weak_countE" (global 140))
2 Likes
34

@berestovskyy suggested an idea of implementing removal of unused globals in ic-wasm when it does the shrink pass.

@chenyan: I guess you’re the owner of ic-wasm? Wdyt about this suggestion?

1 Like
35

Sure. I think the shrink pass already removes the unused globals. It didn’t get removed because they are exported. If we do a pass to remove all exported globals, I think the shrink pass will remove the unused globals for us?

36

Thanks a lot @ulan for all the info. I tried some manual tinkering, converting wasm to wat, then removing all export lines (except the one with the canister function), then turned back to wasm, ran wasm-opt with -Oz flag, then deployed. It deployed successfully, though not sure if i’m doing this right or how to automate this process!
The code is also broken so i’m not sure if it’s something related to the code with ICP or if it’s related to those changes. Is there a way to produce a debugging stack when calling “dfx canister --network local call” to determine where the code broke exactly?

37

Yes, that’s my understanding as well.

Is there a way to produce a debugging stack when calling “dfx canister --network local call” to determine where the code broke exactly?

@abk is implementing backtrace support. It will take several weeks until the feature is release. If you share the Wasm binary and call parameters, abk might be able to run it locally in a patched replica and give you the backtrace.

38

Thanks a lot, that would be of great help.

Kindly find attached the wasm file with removed exports (the debug info was removed after optimizing though, is there a way to get around with keeping the debug info while removing unneeded globals?):

Command to run after deploying:
dfx canister --network local call ICMLPSample train

39

Thank you @ulan , I will have a look at the options to optimize the wasm as part of the build process in icpp-pro.

40

After some investigation, i found out that the problem results after using Oz flag during compilation process. I tried removing all optimizations during compilation and used Oz flag to optimize the wasm using wasm-opt after removing exports, and all ran well.

Thanks a lot for your valuable input. Here’s a Multilayer perceptron Flashlight sample running fully on chain:
dfx canister --network ic call 52uwu-yyaaa-aaaag-aldoa-cai train