Hi everyone,
We’re starting to work on support for returning canister backtraces when a canister traps as part of the Beryllium milestone. Hopefully this improves the dev experience by making it easier to debug canister issues. Here are some simple examples of roughly what it will look like:
Example 1:
Consider the following Rust canister:
fn baz(input: u32) -> u32 {
if input > 10 {
panic!("uh oh");
}
42
}
fn bar_1() -> u32 {
baz(1)
}
fn bar_2() -> u32 {
baz(100)
}
fn bar_3() -> u32 {
baz(10_000)
}
fn foo() -> u32 {
bar_1() + bar_2() + bar_3()
}
#[ic_cdk::query]
fn get_number() -> String {
format!("Got number {}", foo())
}
The current error you see when executing the query is:
IC0503: Canister rwlgt-iiaaa-aaaaa-aaaaa-cai trapped explicitly: Panicked at 'uh oh', src/report_backtrace_backend/src/lib.rs:7:9.
So you can see that we hit the panic in baz, but it may not be obvious which of the three calls to baz triggered the panic.
With backtraces enabled the full error would become:
IC0503: Canister rwlgt-iiaaa-aaaaa-aaaaa-cai trapped explicitly: Panicked at 'uh oh', src/report_backtrace_backend/src/lib.rs:7:9.
Backtrace:
0: ic_cdk::api::trap
1: ic_cdk::printer::set_panic_hook::{{closure}}
2: std::panicking::rust_panic_with_hook
3: std::panicking::begin_panic_handler::{{closure}}
4: std::sys_common::backtrace::__rust_end_short_backtrace
5: rust_begin_unwind
6: core::panicking::panic_fmt
7: report_backtrace_backend::baz
8: report_backtrace_backend::bar_2
9: report_backtrace_backend::foo
10: report_backtrace_backend::get_number_0_::{{closure}}
11: ic_cdk::futures::spawn
12: canister_query get_number
So now you can see that the panic is coming from the call to baz in bar_2 and why bar_2 was called.
Example 2:
The following canister will attempt to perform an out of bounds access and crash:
use std::ptr::write_volatile;
fn baz(input: u32) -> u32 {
unsafe { write_volatile(input as *mut u32, 5) };
42
}
fn bar() -> u32 {
baz(3_000_000_000)
}
fn foo() -> u32 {
bar()
}
#[ic_cdk::query]
fn get_number() -> String {
format!("Got number {}", foo())
}
Currently the error message for this contains no information about where the error occurs:
IC0502: Canister rwlgt-iiaaa-aaaaa-aaaaa-cai trapped: heap out of bounds.
With backtraces we’d be able to see that the oob access occurs in baz:
IC0502: Canister rwlgt-iiaaa-aaaaa-aaaaa-cai trapped: heap out of bounds.
Backtrace:
0: report_backtrace_backend::baz
1: report_backtrace_backend::bar
2: report_backtrace_backend::foo
3: report_backtrace_backend::get_number_0_::{{closure}}
4: ic_cdk::futures::spawn
5: canister_query get_number
Where to View Backtraces [Edit]
The backtraces will always be dumped to the canister logs. Additionally, the error responses to a message will include the backtrace if the caller has sufficient permissions. One option is to return backtraces only to callers that have permission to read the canister logs (via the log visibility setting).
Timeline
The feature will be available for use with local testing in dfx with the Beryllium milestone and on mainnet afterward. Please follow this thread for announcements on the release in dfx and try it out yourself! Especially if you have specific canister bugs where a backtrace might have helped! And if you currently have a bug where a backtrace could be helpful, please send it to me and I’ll get a backtrace back to you before the feature is released.
Limitations
Note that these backtraces will only contain function names (no source code file names on line numbers). This is because the modifications we make to Wasm modules to measure cycles costs make it trickier to determine actual source code locations and it would also require canisters to include full debug info which would make them much larger. If there’s sufficient interest we could investigate supporting source locations as well though.
Privacy
These backtraces will be included in the error response when a message fails and also dumped to the canister’s logs. In order to not expose canister internals, the backtrace will only be incuded in error responses if the caller is a controller of the canister.
Actionable Error Messages
In addition to the backtraces, we’re trying to make it easier to handle canister errors by including links to error documentation along with the error responses. This documentation will include explanations of the error as well as suggestions on how to fix it. Please let us know if you have suggestions on improvents to those docs!