📣 Canbench 0.2.0 Released — Improved Output, CI Integration, and a Breaking Change

Developers
1

Hi everyone,

Over the past few weeks, we’ve made a number of improvements to canbench, the benchmarking tool for IC-based projects. These changes were released incrementally in versions 0.1.12 through 0.1.18, and now we’re releasing 0.2.0, which introduces a breaking change to the public API.

:puzzle_piece: Why We Changed Things

The need for change came from a real challenge in the stable-structures repo, which runs over 100 benchmarks. With so much data, the original canbench output was hard to read — it printed individual results line by line, making it nearly impossible to spot what really changed, example.

:white_check_mark: What’s New

  • Cleaner output. You can now hide individual benchmark results using --hide-results
  • Concise Summary Report. Use --show-summary to get a clean overview showing:
    • Clear final status
    • Count of total benchmarks, regressions, improvements, unchanged, and new benchmarks
    • Change distribution (max, p75, median, p25, min) for metrics instructions, heap_increase, stable_memory_increase
  • Significant Changes Table. See the top 50 biggest changes
  • Full CSV Report. Use --csv to export the full benchmark data into a CSV file, which is also uploaded as a CI artifact so you can explore it with tools like Google Sheets (e.g., for postprocessing, filtering, formatting).
  • GitHub CI Integration. Updated CI scripts now:
    • (as before) Post a short summary (and optional full output) as a comment in your PR, see example
    • Include commit hash and timestamp, so you know how fresh the report is
    • Attach CSV files to the CI run as downloadable artifacts

:hammer_and_wrench: API Change in 0.2.0

We introduced a new field pub calls: u64 to Measurement structure.

This tracks how many times a given benchmark scope was executed. This is especially useful for repeated or recursive scopes — even if a single run is cheap, being called thousands of times might make it a performance hotspot.

Note: this tracking adds some overhead, especially in hot paths, so it’s good to keep an eye on calls to avoid misinterpreting results.

:repeat_button: How to Upgrade

  • Bump canbench version to 0.2.0
  • Use --locked to avoid surprises from indirect dependency updates
    • When installing
      $ cargo install --version 0.2.0 --locked canbench, example
    • When building with build_cmd in your canbench.yml, example
  • Update your benchmark running scripts (examples are available in canbench and stable-structures) and corresponding CI jobs to upload CSV file, example

:warning: Note: if your CI compares benchmarks on main vs PR branch — and your main branch still uses 0.1.x — expect a one-time failure due to the breaking API change. Once merged, this will be resolved.

:package: Migration Example Repository

We’ve published a small demo repo to help you with the transition:

:backhand_index_pointing_right: GitHub - dfinity/canbench-0.2.0-demo: canbench 0.1.11 → 0.2.0 migration demo

This example shows:

  • The previous output format using canbench 0.1.11
  • The migration to 0.2.0
  • The new output format and CI integration

This not only illustrates the changes but also provides a concrete reference for migrating your own project.

Thanks for using canbench, and feel free to open issues or suggestions — your real-world examples help us make the tool better!

8 Likes
2

Just added a new section to the post with a link to a demo repo showing the migration from canbench 0.1.11 to 0.2.0.

It includes both old and new outputs to help you compare and upgrade more easily.

Check it out!

3

In 0.2.1, is the entire function body automatically timed making it impossible to exclude setup code from measurements? Previously, I tested subcomponents individually by wrapping them in
canbench_rs::bench_fn(|| {()} {} ) and found it really useful.

4

I don’t think that’s true. We are using canbench@0.2.1 for NNS canisters, and #[bench(raw)] (with bench_fn inside) can still measure a particular part of the benchmark, excluding the setup code. The 0.2.1 documentation also still references the “excluding setup code” section.

1 Like