📣 Canbench 0.2.0 Released — Improved Output, CI Integration, and a Breaking Change

Developers
1

Hi everyone,

Over the past few weeks, we’ve made a number of improvements to canbench, the benchmarking tool for IC-based projects. These changes were released incrementally in versions 0.1.12 through 0.1.18, and now we’re releasing 0.2.0, which introduces a breaking change to the public API.

:puzzle_piece: Why We Changed Things

The need for change came from a real challenge in the stable-structures repo, which runs over 100 benchmarks. With so much data, the original canbench output was hard to read — it printed individual results line by line, making it nearly impossible to spot what really changed, example.

:white_check_mark: What’s New

  • Cleaner output. You can now hide individual benchmark results using --hide-results
  • Concise Summary Report. Use --show-summary to get a clean overview showing:
    • Clear final status
    • Count of total benchmarks, regressions, improvements, unchanged, and new benchmarks
    • Change distribution (max, p75, median, p25, min) for metrics instructions, heap_increase, stable_memory_increase
  • Significant Changes Table. See the top 50 biggest changes
  • Full CSV Report. Use --csv to export the full benchmark data into a CSV file, which is also uploaded as a CI artifact so you can explore it with tools like Google Sheets (e.g., for postprocessing, filtering, formatting).
  • GitHub CI Integration. Updated CI scripts now:
    • (as before) Post a short summary (and optional full output) as a comment in your PR, see example
    • Include commit hash and timestamp, so you know how fresh the report is
    • Attach CSV files to the CI run as downloadable artifacts

:hammer_and_wrench: API Change in 0.2.0

We introduced a new field pub calls: u64 to Measurement structure.

This tracks how many times a given benchmark scope was executed. This is especially useful for repeated or recursive scopes — even if a single run is cheap, being called thousands of times might make it a performance hotspot.

Note: this tracking adds some overhead, especially in hot paths, so it’s good to keep an eye on calls to avoid misinterpreting results.

:repeat_button: How to Upgrade

  • Bump canbench version to 0.2.0
  • Use --locked to avoid surprises from indirect dependency updates
    • When installing
      $ cargo install --version 0.2.0 --locked canbench, example
    • When building with build_cmd in your canbench.yml, example
  • Update your benchmark running scripts (examples are available in canbench and stable-structures) and corresponding CI jobs to upload CSV file, example

Note: if your CI compares benchmarks on main vs PR branch — and your main branch still uses 0.1.x — expect a one-time failure due to the breaking API change. Once merged, this will be resolved.

Thanks for using canbench, and feel free to open issues or suggestions — your real-world examples help us make the tool better!

6 Likes