Introducing "canbench": a benchmarking framework for canisters

Developers Programs & Applications
1

Hey everyone!

I wanted to share with you a little tool that we’ve been working on and just open-sourced. It’s called canbench and it’s a benchmarking framework for canisters.

canbench is a tool that we built organically when we were trying to understand the performance of canisters and libraries that we were building. Once setup, it allows you to effortlessly monitor performance regressions as well as analyze the performance of your code.

Here is an example of what the canbench CI integration looks like (screenshots below):

Performance changes are detected:

image
image1834×1294 223 KB

No performance changes are detected:

image
image1818×1180 175 KB

Currently canbench only supports Rust canisters, but support for other languages can easily be introduced. Let us know if this is something you’d like to see. Happy benching :weight_lifting_man:

13 Likes
2

Hey @ielashi,

this is awesome, and as such, should be added to GitHub - dfinity/awesome-internet-computer: A curated list of awesome projects and resources relating to the Internet Computer Protocol :wink:

Would you mind doing a PR?

2 Likes
3

Anyone got any ideas of what might be causing this error?

“error parsing result of benchmark test. Result: module”

1 Like
4

Sure, will do :slight_smile:

It looks like there’s an error when executing the benchmark. Can you share the benchmark code and/or wasm module that’s causing this error?

1 Like
5

I am getting an error when trying to run canbench

Error executing benchmark create_account_bench. Error:
ingress Completed: Reply: 0x4449444c016c01b3c4b1f204680100010a00000000000000000101
ingress Err: IC0503: Canister rwlgt-iiaaa-aaaaa-aaaaa-cai trapped explicitly: failed to decode call arguments: Custom(Cannot parse header 

Caused by:
    binary parser error: io error)
Err: IC0304: Attempt to execute a message on canister rwlgt-iiaaa-aaaaa-aaaaa-cai which contains no Wasm module
2 Likes
6

Apologies for the slow response here. Is there code you can point me to to triage the problem? My github username is “ielashi” in case you’d like to share access to your code.

1 Like
7

repo

1 Like
8

Hi @ielashi, Do you know why I am getting this error?

1 Like
9

@zohaib29 Is your canister taking in init args? canbench doesn’t currently support deploying a wasm that has init args. If yes, can you try removing them and see if that fixes the issue?

We should be able to add support for specifying the init args in canbench.yml - it’s not that much work.

2 Likes
10

Yes, after removing init args it worked. Thanks!

1 Like
11

Thanks for the feedback. I added two issues to improve canbench in that regard.

2 Likes
12

Update: The newest version of canbench (version 0.1.4) now supports canisters with init args. See here for the documentation. Big thanks to @frederikrothenberger for adding this feature!

cc @zohaib29

3 Likes
13

I am attempting to integrate canbench into this open-source repo for running Candle based models within canisters. I am currently working on integrating it into this branch at this exact position. It is unclear to me how I would upload the model weights as part of the canbench procedure. I have tried several approaches. Hopefully, the authors (@ielashi et al) can provide some insight into how best to go about loading large files which impact instruction counts and heap utilization.

1 Like
14

@jeshli I was running into this exact problem as well. I added a feature a few weeks ago to allow you to specify a file to be loaded in stable memory before running the benchmarks here. Let me add a few polishes and create a new canbench release.

1 Like
15

This is great. I built it locally because I was too excited to wait. It’s working great. I put my results front and center in the open-source GPT2-example README. More to come in that regard.

1 Like
16

@ielashi ,

I am interested to get canbench to work for C++ projects where the wasm is build using icpp-pro.

To get started, I forked canbench, and created the fibonacci_cpp example, which is functional:

Next, I want to explore the best way to get the canbench instrumentations into the wasm.

My first thought is to use a similar approach as the way we include the WasiPolyfill in the C++ build, by linking in a static rust library. I am not sure if that is possible, but that is my first thought. To see how it is done with wasi polyfill, see examples/hello-world-cpp at main · wasm-forge/examples · GitHub

@sgaflv , since you created the wasi polyfill and showed how to include that in a C++ based wasm, let me tag you as well. If any suggestions, that would be highly appreciated.

17

Any luck with the linking approach?