Canister Backtraces

Developers
21

Canister backtraces aren’t working in Azle, I keep getting errors like this (dfx 0.24.3):

dfx canister call async_await test
Error: Failed update call.
Caused by: The replica returned a rejection error: reject code CanisterError, reject message Error from Canister bkyz2-fmaaa-aaaaa-qaaaq-cai: Canister called `ic0.trap` with message: Uncaught TypeError: Error converting from js 'i32' into type 'u8': Overflow
    at StableBTreeMap (.azle/async_await/main.js:5519:53)
    at test (.azle/async_await/main.js:5758:19)
    at getUnencodedResult (.azle/async_await/main.js:5069:18)
    at executeAndReplyWithCandidSerde (.azle/async_await/main.js:5030:5)
    at <anonymous> (.azle/async_await/main.js:5275:13)

Canister Backtrace:
unknown function at index 2927
unknown function at index 1734
unknown function at index 350
unknown function at index 2695
unknown function at index 46
unknown function at index 46
unknown function at index 46
unknown function at index 46
unknown function at index 363
unknown function at index 1737
unknown function at index 46
unknown function at index 993
unknown function at index 1398
unknown function at index 1038
unknown function at index 116
unknown function at index 3257

We don’t use any Wasm optimization that I know of that would strip the name metadata section from the Wasm binary. We are using a dfx extension. I’ve tried adding back debug = true to our release profile in Cargo.toml, but that doesn’t seem to have any effect.

The name section doesn’t appear to be included:

wasm-objdump -h .azle/async_await/async_await.wasm 

async_await.wasm:	file format wasm 0x1

Sections:

     Type start=0x0000000b end=0x000004c7 (size=0x000004bc) count: 147
   Import start=0x000004ca end=0x00000881 (size=0x000003b7) count: 41
 Function start=0x00000884 end=0x00001545 (size=0x00000cc1) count: 3244
    Table start=0x00001547 end=0x0000154e (size=0x00000007) count: 1
   Memory start=0x00001550 end=0x00001553 (size=0x00000003) count: 1
   Global start=0x00001555 end=0x000015a2 (size=0x0000004d) count: 11
   Export start=0x000015a5 end=0x00001d38 (size=0x00000793) count: 69
     Elem start=0x00001d3b end=0x0000254d (size=0x00000812) count: 1
DataCount start=0x0000254f end=0x00002550 (size=0x00000001) count: 4
     Code start=0x00002555 end=0x002150e3 (size=0x00212b8e) count: 3244
     Data start=0x002150e7 end=0x0032e8b7 (size=0x001197d0) count: 4
   Custom start=0x0032e8ba end=0x0032e980 (size=0x000000c6) "producers"
   Custom start=0x0032e982 end=0x0032e9d8 (size=0x00000056) "target_features"

Here’s our extension.json for the azle extension:

{
    "name": "azle",
    "version": "0.26.0",
    "homepage": "https://github.com/dfinity/dfx-extensions",
    "authors": "",
    "summary": "",
    "categories": [],
    "keywords": [],
    "canister_type": {
        "defaults": {
            "candid": ".azle/{{canister_name}}/{{canister_name}}.did",
            "build": "node_modules/.bin/azle compile {{canister_name}}",
            "post_install": "node_modules/.bin/azle upload-assets {{canister_name}}",
            "wasm": ".azle/{{canister_name}}/{{canister_name}}.wasm",
            "gzip": true,
            "metadata": [
                {
                    "name": "candid:service"
                },
                {
                    "name": "cdk:name",
                    "content": "azle"
                }
            ],
            "tech_stack": {
                "cdk": {
                    "azle": {}
                },
                "language": {
                    "typescript": {},
                    "javascript": {}
                }
            }
        }
    }
}
22

This is how we compile default Rust canisters. With that you should get the name section. dfx also doesn’t minimize the wasm unless you ask it to :person_shrugging:

23

About actually getting the backtraces working, I see a few steps in the azle build which might be stripping out the name section and I think you need to look at the artifacts after each of those steps to see if the name section is still there:

  1. The experimental.wasm and stable.wasm templates in the repo do have name sections. If the build process for an azle canister uses these directly then the starting point is good, but if they get rebuilt I’d double check that they still have the name section.
  2. The dfx extension seems to run your azle compile command to add the passive data section (and other stuff) using binaryen. It’s possible this step could be implicitly removing the name section. I would expect binaryen to try to keep the name section, but maybe it isn’t in this case.
  3. I think the metadata section of the dfx extension will cause dfx to run an ic-wasm metadata step on your Wasm before installing it. It’s possible this command isn’t being run with the --keep-name-section flag by default and the extension needs to be modified to include it.

Once we know which of these steps is stripping the name section it will be easier to fix.

But at a higher level, I’m wondering if canister backtraces really make sense for interpreted languages like azle and kybra. Your example above already contains the backtrace from the canister author’s JS code which is probably what they want to see. The missing function names are going to show where in the JS engine the error is encountered so I think it might be helpful for you and your team when debugging issues in azle itself, but it’s probably not useful for a user of azle. So I’d imagine you might want the option of canister backtraces for your team’s work, but they should probably not be included in the production system.

1 Like
24

This sounds right. We’ve done a lot of work on our error messages and now the backtraces are kind of a step backwards. Being able to have them on during development might be helpful, but turning them off I think is most important.

25

@abk Who can I talk to about the ability to disable canister backtraces? I think we’d really like to toggle that off unless in a development mode.

26

I also wanted to give an update on this, it looks like binaryen.readBinary is removing the name section. I have opened an issue with binaryen.js to try and resolve this: The name section is removed by `binaryen.readBinary` · Issue #99 · AssemblyScript/binaryen.js · GitHub

27

I think the cleanest solution would be to just remove the backtrace when the name section isn’t included and then you can use that to toggle it. It would be a quick change.

1 Like
28

Awesome, this will actually work perfectly. I just discovered a function binaryen.setDebuginfo which keeps the name section in the final binary, so it will be dead simple for us to toggle.

29

Do you have an open issue I can follow? Or an ETA or expected version of dfx this change will be in?

This is actually a pretty material change that will make Azle’s errors much easier to read for end developers, if looks even worse when the backtrace is already there :slight_smile: Though Azle core devs will enjoy this.

30

I should be able to get it done this week, then it’ll be rolled out on mainnet next week.

2 Likes
31

That would be excellent!