# Introducing "canbench": a benchmarking framework for canisters **URL:** \ **Category:** Programs & Applications\ **Created:** [February 12, 2024, 1:42pm UTC](https://forum.dfinity.org/t/introducing-canbench-a-benchmarking-framework-for-canisters/27517 "2024-02-12T13:42:24Z")\ **Posts on this page:** 17\ **Page:** 1
**Author:** ![ielashi](https://sea1.discourse-cdn.com/flex023/user_avatar/forum.dfinity.org/ielashi/32/15704_2.png) [@ielashi](https://forum.dfinity.org/u/ielashi)\ **Post date:** [February 12, 2024, 1:42pm UTC](https://forum.dfinity.org/t/introducing-canbench-a-benchmarking-framework-for-canisters/27517/1 "2024-02-12T13:42:25Z")
Hey everyone! I wanted to share with you a little tool that we’ve been working on and just open-sourced. It’s called [canbench](https://docs.rs/canbench-rs/latest/canbench_rs/) and it’s a benchmarking framework for canisters. `canbench` is a tool that we built organically when we were trying to understand the performance of canisters and libraries that we were building. Once setup, it allows you to effortlessly monitor performance regressions as well as analyze the performance of your code. Here is an [example](https://github.com/dfinity/canbench/pull/25) of what the `canbench` CI integration looks like (screenshots below): Performance changes are detected: ![image](https://us1.discourse-cdn.com/flex023/uploads/dfn/original/2X/b/bfa8be39d96cfcf781f4466ae5702c8b221ab61b.png) No performance changes are detected: ![image](https://us1.discourse-cdn.com/flex023/uploads/dfn/original/2X/a/a5c4384102a9377d37d2d09670bbccbfc9449836.png) Currently `canbench` only supports Rust canisters, but support for other languages can easily be introduced. Let us know if this is something you’d like to see. Happy benching 🏋️‍♂️ ---
**Author:** ![domwoe](https://sea1.discourse-cdn.com/flex023/user_avatar/forum.dfinity.org/domwoe/32/5165_2.png) [@domwoe](https://forum.dfinity.org/u/domwoe)\ **Post date:** [February 12, 2024, 7:25pm UTC](https://forum.dfinity.org/t/introducing-canbench-a-benchmarking-framework-for-canisters/27517/2 "2024-02-12T19:25:22Z")
Hey @ielashi, this is awesome, and as such, should be added to [GitHub - dfinity/awesome-internet-computer: A curated list of awesome projects and resources relating to the Internet Computer Protocol](https://github.com/dfinity/awesome-internet-computer) 😉 Would you mind doing a PR? ---
**Author:** ![NS01](https://sea1.discourse-cdn.com/flex023/user_avatar/forum.dfinity.org/ns01/32/9816_2.png) [@NS01](https://forum.dfinity.org/u/NS01)\ **Post date:** [February 24, 2024, 6:55pm UTC](https://forum.dfinity.org/t/introducing-canbench-a-benchmarking-framework-for-canisters/27517/3 "2024-02-24T18:55:10Z")
Anyone got any ideas of what might be causing this error? “error parsing result of benchmark test. Result: module” ---
**Author:** ![ielashi](https://sea1.discourse-cdn.com/flex023/user_avatar/forum.dfinity.org/ielashi/32/15704_2.png) [@ielashi](https://forum.dfinity.org/u/ielashi)\ **Post date:** [February 27, 2024, 7:39am UTC](https://forum.dfinity.org/t/introducing-canbench-a-benchmarking-framework-for-canisters/27517/4 "2024-02-27T07:39:42Z")
> [@domwoe](#): > > this is awesome, and as such, should be added to [GitHub - dfinity/awesome-internet-computer: A curated list of awesome projects and resources relating to the Internet Computer Protocol](https://github.com/dfinity/awesome-internet-computer) 😉 > > Would you mind doing a PR? Sure, will do 🙂 > [@NS01](#): > > Anyone got any ideas of what might be causing this error? > > “error parsing result of benchmark test. Result: module” It looks like there’s an error when executing the benchmark. Can you share the benchmark code and/or wasm module that’s causing this error? ---
**Author:** ![zohaib29](https://sea1.discourse-cdn.com/flex023/user_avatar/forum.dfinity.org/zohaib29/32/29625_2.png) [@zohaib29](https://forum.dfinity.org/u/zohaib29)\ **Post date:** [April 26, 2024, 6:06pm UTC](https://forum.dfinity.org/t/introducing-canbench-a-benchmarking-framework-for-canisters/27517/5 "2024-04-26T18:06:41Z")
I am getting an error when trying to run `canbench` ```auto Error executing benchmark create_account_bench. Error: ingress Completed: Reply: 0x4449444c016c01b3c4b1f204680100010a00000000000000000101 ingress Err: IC0503: Canister rwlgt-iiaaa-aaaaa-aaaaa-cai trapped explicitly: failed to decode call arguments: Custom(Cannot parse header Caused by: binary parser error: io error) Err: IC0304: Attempt to execute a message on canister rwlgt-iiaaa-aaaaa-aaaaa-cai which contains no Wasm module ``` ---
**Author:** ![ielashi](https://sea1.discourse-cdn.com/flex023/user_avatar/forum.dfinity.org/ielashi/32/15704_2.png) [@ielashi](https://forum.dfinity.org/u/ielashi)\ **Post date:** [April 29, 2024, 8:24am UTC](https://forum.dfinity.org/t/introducing-canbench-a-benchmarking-framework-for-canisters/27517/6 "2024-04-29T08:24:13Z")
Apologies for the slow response here. Is there code you can point me to to triage the problem? My github username is “ielashi” in case you’d like to share access to your code. ---
**Author:** ![zohaib29](https://sea1.discourse-cdn.com/flex023/user_avatar/forum.dfinity.org/zohaib29/32/29625_2.png) [@zohaib29](https://forum.dfinity.org/u/zohaib29)\ **Post date:** [April 29, 2024, 9:46am UTC](https://forum.dfinity.org/t/introducing-canbench-a-benchmarking-framework-for-canisters/27517/7 "2024-04-29T09:46:32Z")
[repo](https://github.com/AegisFinance/aegis-accounts/blob/c15851b7009fa6d21bec1afd814e84af5ac40d2a/src/canisters/accounts/src/queries/account.rs#L13) ---
**Author:** ![zohaib29](https://sea1.discourse-cdn.com/flex023/user_avatar/forum.dfinity.org/zohaib29/32/29625_2.png) [@zohaib29](https://forum.dfinity.org/u/zohaib29)\ **Post date:** [April 30, 2024, 7:20am UTC](https://forum.dfinity.org/t/introducing-canbench-a-benchmarking-framework-for-canisters/27517/8 "2024-04-30T07:20:56Z")
Hi @ielashi, Do you know why I am getting this error? ---
**Author:** ![ielashi](https://sea1.discourse-cdn.com/flex023/user_avatar/forum.dfinity.org/ielashi/32/15704_2.png) [@ielashi](https://forum.dfinity.org/u/ielashi)\ **Post date:** [April 30, 2024, 8:33am UTC](https://forum.dfinity.org/t/introducing-canbench-a-benchmarking-framework-for-canisters/27517/9 "2024-04-30T08:33:33Z")
@zohaib29 Is your canister taking in init args? `canbench` doesn’t currently support deploying a wasm that has init args. If yes, can you try removing them and see if that fixes the issue? We should be able to add support for specifying the init args in `canbench.yml` - it’s not that much work. ---
**Author:** ![zohaib29](https://sea1.discourse-cdn.com/flex023/user_avatar/forum.dfinity.org/zohaib29/32/29625_2.png) [@zohaib29](https://forum.dfinity.org/u/zohaib29)\ **Post date:** [April 30, 2024, 8:56am UTC](https://forum.dfinity.org/t/introducing-canbench-a-benchmarking-framework-for-canisters/27517/10 "2024-04-30T08:56:17Z")
Yes, after removing init args it worked. Thanks! ---
**Author:** ![ielashi](https://sea1.discourse-cdn.com/flex023/user_avatar/forum.dfinity.org/ielashi/32/15704_2.png) [@ielashi](https://forum.dfinity.org/u/ielashi)\ **Post date:** [May 2, 2024, 7:18am UTC](https://forum.dfinity.org/t/introducing-canbench-a-benchmarking-framework-for-canisters/27517/11 "2024-05-02T07:18:00Z")
Thanks for the feedback. I added two issues to improve `canbench` in that regard. > > > If canbench fails to install the benchmarking canister, it currently returns a c…ryptic error: > > \`\`\` > Error executing benchmark create\_account\_bench. Error: > ingress Completed: Reply: 0x4449444c016c01b3c4b1f204680100010a00000000000000000101 > ingress Err: IC0503: Canister rwlgt-iiaaa-aaaaa-aaaaa-cai trapped explicitly: failed to decode call arguments: Custom(Cannot parse header > > Caused by: > binary parser error: io error) > Err: IC0304: Attempt to execute a message on canister rwlgt-iiaaa-aaaaa-aaaaa-cai which contains no Wasm module > \`\`\` > > canbench should return a more explicit error message that states: > > 1. The canister installation failed. > 2. Why it failed. > > > \`canbench\` currently can only deploy a canister to benchmark if that canister ta…kes no arguments. It'd be desirable for many users to be able to specify some arguments. Perhaps an option to specify the args can be added to \`canbench.yml\`. ---
**Author:** ![ielashi](https://sea1.discourse-cdn.com/flex023/user_avatar/forum.dfinity.org/ielashi/32/15704_2.png) [@ielashi](https://forum.dfinity.org/u/ielashi)\ **Post date:** [May 21, 2024, 1:12pm UTC](https://forum.dfinity.org/t/introducing-canbench-a-benchmarking-framework-for-canisters/27517/12 "2024-05-21T13:12:42Z")
Update: The newest version of canbench (version 0.1.4) now supports canisters with init args. See [here](https://docs.rs/canbench-rs/latest/canbench_rs/#init-args) for the documentation. Big thanks to @frederikrothenberger for adding this feature! cc @zohaib29 ---
**Author:** ![jeshli](https://sea1.discourse-cdn.com/flex023/user_avatar/forum.dfinity.org/jeshli/32/19879_2.png) [@jeshli](https://forum.dfinity.org/u/jeshli)\ **Post date:** [January 13, 2025, 3:20pm UTC](https://forum.dfinity.org/t/introducing-canbench-a-benchmarking-framework-for-canisters/27517/13 "2025-01-13T15:20:09Z")
I am attempting to integrate canbench into this open-source repo for running Candle based models within canisters. I am currently working on integrating it into [this branch](https://github.com/decide-ai/decide-ai-ic/tree/jeshli/MOD-1694/canbench) at this [exact position](https://github.com/decide-ai/decide-ai-ic/blob/fa4a5211192a76bc908c91949c6d9ed0a0870671/examples/gpt2/src/model/src/llm/candle.rs#L166). It is unclear to me how I would upload the model weights as part of the canbench procedure. I have tried several approaches. Hopefully, the authors (@ielashi et al) can provide some insight into how best to go about loading large files which impact instruction counts and heap utilization. ---
**Author:** ![ielashi](https://sea1.discourse-cdn.com/flex023/user_avatar/forum.dfinity.org/ielashi/32/15704_2.png) [@ielashi](https://forum.dfinity.org/u/ielashi)\ **Post date:** [January 17, 2025, 10:45am UTC](https://forum.dfinity.org/t/introducing-canbench-a-benchmarking-framework-for-canisters/27517/14 "2025-01-17T10:45:04Z")
@jeshli I was running into this exact problem as well. I added a feature a few weeks ago to allow you to specify a file to be loaded in stable memory before running the benchmarks [here](https://github.com/dfinity/canbench/pull/76). Let me add a few polishes and create a new canbench release. ---
**Author:** ![jeshli](https://sea1.discourse-cdn.com/flex023/user_avatar/forum.dfinity.org/jeshli/32/19879_2.png) [@jeshli](https://forum.dfinity.org/u/jeshli)\ **Post date:** [January 17, 2025, 9:13pm UTC](https://forum.dfinity.org/t/introducing-canbench-a-benchmarking-framework-for-canisters/27517/15 "2025-01-17T21:13:19Z")
This is great. I built it locally because I was too excited to wait. It’s working great. I put my results front and center in the open-source GPT2-example [README](https://github.com/decide-ai/decide-ai-ic/tree/main/examples/gpt2). More to come in that regard. ---
**Author:** ![icpp](https://sea1.discourse-cdn.com/flex023/user_avatar/forum.dfinity.org/icpp/32/9310_2.png) [@icpp](https://forum.dfinity.org/u/icpp)\ **Post date:** [February 3, 2025, 9:19pm UTC](https://forum.dfinity.org/t/introducing-canbench-a-benchmarking-framework-for-canisters/27517/16 "2025-02-03T21:19:49Z")
@ielashi , I am interested to get canbench to work for C++ projects where the wasm is build using icpp-pro. To get started, I forked canbench, and created the `fibonacci_cpp` example, which is functional: > **[canbench/examples/fibonacci\_cpp at fibonacci\_cpp · icppWorld/canbench](https://github.com/icppWorld/canbench/tree/fibonacci_cpp/examples/fibonacci_cpp)** > > A benchmarking framework for canisters on the Internet Computer. - icppWorld/canbench Next, I want to explore the best way to get the canbench instrumentations into the wasm. My first thought is to use a similar approach as the way we include the WasiPolyfill in the C++ build, by linking in a static rust library. I am not sure if that is possible, but that is my first thought. To see how it is done with wasi polyfill, see [examples/hello-world-cpp at main · wasm-forge/examples · GitHub](https://github.com/wasm-forge/examples/tree/main/hello-world-cpp) @sgaflv , since you created the wasi polyfill and showed how to include that in a C++ based wasm, let me tag you as well. If any suggestions, that would be highly appreciated. ---
**Author:** ![ielashi](https://sea1.discourse-cdn.com/flex023/user_avatar/forum.dfinity.org/ielashi/32/15704_2.png) [@ielashi](https://forum.dfinity.org/u/ielashi)\ **Post date:** [February 7, 2025, 10:47am UTC](https://forum.dfinity.org/t/introducing-canbench-a-benchmarking-framework-for-canisters/27517/17 "2025-02-07T10:47:47Z")
Any luck with the linking approach?