Performance testing best practices

Developers
1

It would be helpful to collect resources on how to best do performance testing, measurements and benchmarking for canister code and provide that collection to newcomers in one place. I don’t have anything to offer myself on this topic so I am asking anyone who has experience to share their approaches and best practices.

One way to categorize the approaches would be by the tools required. For example, say I have only moc and wastime and some Motoko code (or rustc and wasmtime). Is there meaningful performance testing that I can do with that? If yes, how?

The next approach, if we have dfx available, is to start a local replica and use the IC’s performance counter. Are there any best practices on how to use it for systematic benchmarking? How best to integrate a benchmark into CI? In case of Motoko how to include the garbage collection into measurements taken with the performance counter?

The Motoko playground has profiling and flamegraphs. Are there resources on how to do the same from the command line?

Are there already existing benchmarking frameworks on github?

5 Likes
2

I’d like to see something like this in dfx. After each call:

  • display the number of wasm instructions executed

  • display the (in/dec)crease in storage after an update call. this might be more difficult since we might be counting memory pages instead of total bytes.

3

This might (haven’t tested) be available if you set the replica to debug log level: sdk/CHANGELOG.md at master · dfinity/sdk · GitHub (requires latest dfx master, this is not (yet) available in a normal release)

4

Is there a single place where performance utils are listed now?

Should we make one?

I would add it to portal if you throw all tools at me here