Azle/Motoko/Rust benchmarks

Developers
1

At Demergent Labs we’ve been spending the last few weeks working on an automated benchmarking process that would allow us to compare Wasm instruction usage and ultimately cycle costs in USD for applications written in Azle (TypeScript), Motoko, and Rust. We have created the initial benchmarking framework and have run the benchmarks on a number of examples written in Azle, Motoko, and Rust.

The results should be considered rough, and there are most likely various flaws/simplifications in our process. But I think the results are instructive. I’m including the relevant section from Azle’s documentation below, it should have a lot of useful information and links to follow for more info.

The primitive_ops example may be of particular interest, as it has specific benchmarks for operating on nearly all Candid types across the different languages.

I’m soliciting feedback on our benchmarking process. I’m specifically concerned that ic0.performance_counter is not accurately measuring Motoko Wasm instructions, as you may be able to notice from our results. Of course we could be much more rigorous with our statistics, and charts and graphs and various metrics could be added to improve these results. We’re mostly interested in being able to give developers a rough but accurate estimate of the trade-offs in bottom-line costs when deciding to use Azle over Rust and Motoko.

Benchmarks

TLDR

Last major update to this benchmarking information: August 2022

Here’s a quick rough estimate summary of the benchmarks:

  • Average/Median Azle Wasm instructions per function call: 121_919_698 / 2_387_322

  • Average/Median Motoko Wasm instructions per function call: 272_877 / 5_169

  • Average/Median Rust Wasm instructions per function call: 26_378_678 / 44_074

  • Average/Median Azle/Motoko Wasm instructions difference: 1_273x / 396x

  • Average/Median Azle/Rust Wasm instructions difference: 204x / 51x

  • Average/Median Motoko/Rust Wasm instructions difference: -22x / -8x

  • Average/Median Azle/Motoko USD cost estimate difference using average Wasm instructions: 5x / 3x

  • Maximum Azle/Motoko USD cost estimate difference using average Wasm instructions: 13x

  • Average/Median Azle/Rust USD cost estimate difference using average Wasm instructions: 2x / 2x

  • Maximum Azle/Rust USD cost estimate difference using average Wasm instructions: 4x

  • Average/Median Azle/Motoko USD cost difference using median Wasm instructions: 1x / 1x

  • Maximum Azle/Motoko USD cost difference using median Wasm instructions: 1x

  • Average/Median Azle/Rust USD cost difference using median Wasm instructions: 1x / 1x

  • Maximum Azle/Rust USD cost difference using median Wasm instructions: 1x

Azle’s automated benchmarking framework is currently based on the ic0.performance_counter System API. ic0.performance_counter seems to have a number of limitations. The following may not be measured accurately:

  • Candid serialization/deserialization of function parameters and return types
  • Canister method prologue/epilogue
  • Some Motoko runtime behavior (such as garbage collection and method prologue)

You can find out more information about ic0.performance_counter here.

We currently have benchmarks for the following examples written with Azle, Motoko, and Rust:

Below are the results of aggregating all of the benchmarking key metrics from the examples above.

function body only means measurements were taken only from the beginning to the end of the function body of the function under measurement. function prelude and body means measurements were taken at the end of the body of the function under measurement, including what may have happend before the function was invoked. It may be more realistic to rely on function prelude and body, as it captures more of the real Wasm instruction usage. That being said, something seems off with the Motoko prelude measurements.

Wasm Instructions

The number of Wasm instructions counted by ic0.performance_counter for a function under measurement. These do not translate directly into actual cycle costs. See the section on USD cost estimates below for more information.

Azle
  • function body only
    • Average: 90_880_192
    • Median: 1_974_800
  • function prelude and body
    • Average: 121_919_698
    • Median: 2_387_322
Motoko
  • function body only
    • Average: 225_577
    • Median: 3_361
  • function prelude and body
    • Average: 272_877
    • Median: 5_169
Rust
  • function body only
    • Average: 88_912
    • Median: 2_820
  • function prelude and body
    • Average: 26_378_678
    • Median: 44_074

Change Multipliers

Change multipliers show how many times greater or fewer Wasm instructions are used when comparing one language to another. A positive number means more instructions are used by language a vs language b. A negative number means fewer instructions are used by language a vs language b. Languages a and b are determined with this pattern: considering Azle/Motoko, Azle is language a and Motoko is language b.

Azle/Motoko
  • function body only
    • Average: 2_649x
    • Median: 468x
  • function prelude and body
    • Average: 1_273x
    • Median: 396x
Azle/Rust
  • function body only
    • Average: 2_426x
    • Median: 233x
  • function prelude and body
    • Average: 204x
    • Median: 51x
Motoko/Rust
  • function body only:
    • Average: -3x
    • Median: 1x
  • function prelude and body:
    • Average: -22x
    • Median: -8x

USD Cost Estimates

An attempt to estimate actual USD costs per year based on the Wasm instruction counts. A number of application scenarios are presented that attempt to simulate real-world application usage. Cycle costs are taken from here.

Average

These estimates use the average Wasm instructions per function call including the function prelude.

The Wasm instruction counts used are:

  • Azle: 121_919_698
  • Motoko: 272_877
  • Rust: 26_378_678
Cycle Costs Table

Cycle costs taken from here.

Compute Percent Allocated Per Second Update Message Execution Ten Update Instructions Execution Xnet Call Xnet Byte Transmission Ingress Message Reception Ingress Byte Reception GB Storage Per Second
100_000 590_000 4 260_000 1_000 1_200_000 2_000 127_000
Application Scenarios
Usage Query/Update Heaviness Ingress Bytes Per Query Message Ingress Bytes Per Update Message GB Storage Query Messages Per Second Update Messages Per Second Xnet Calls Per Second Xnet Call Bytes
Light Even 100 100 0.5 0.01 0.01 0.001 20
Light Query Heavy 100 100 0.5 0.01 0.0001 0.001 20
Light Update Heavy 100 100 0.5 0.0001 0.01 0.001 20
Moderate Even 1_000 1_000 1 1 1 0.1 200
Moderate Query Heavy 1_000 1_000 1 1 0.01 0.1 200
Moderate Update Heavy 1_000 1_000 1 0.01 1 0.1 200
Heavy Even 10_000 10_000 2 100 100 10 2_000
Heavy Query Heavy 10_000 10_000 2 100 1 10 2_000
Heavy Update Heavy 10_000 10_000 2 1 100 10 2_000
Application USD Cost Estimates Per Year
Usage Query/Update Heaviness CDK Ingress Messages Ingress Bytes Query Messages Ingress Bytes Update Messages Update Messages Update Instructions Xnet Calls Xnet Byte Transmission GB Storage Total Cost
Light Even Azle $1.00 $0.08 $0.08 $0.25 $20.30 $0.01 $0.00 $2.64 $24.37
Light Even Motoko $1.00 $0.08 $0.08 $0.25 $0.05 $0.01 $0.00 $2.64 $4.11
Light Even Rust $1.00 $0.08 $0.08 $0.25 $4.39 $0.01 $0.00 $2.64 $8.46
Light Query Heavy Azle $0.50 $0.08 $0.00 $0.00 $0.20 $0.01 $0.00 $2.64 $3.45
Light Query Heavy Motoko $0.50 $0.08 $0.00 $0.00 $0.00 $0.01 $0.00 $2.64 $3.25
Light Query Heavy Rust $0.50 $0.08 $0.00 $0.00 $0.04 $0.01 $0.00 $2.64 $3.29
Light Update Heavy Azle $0.50 $0.00 $0.08 $0.25 $20.30 $0.01 $0.00 $2.64 $23.79
Light Update Heavy Motoko $0.50 $0.00 $0.08 $0.25 $0.05 $0.01 $0.00 $2.64 $3.53
Light Update Heavy Rust $0.50 $0.00 $0.08 $0.25 $4.39 $0.01 $0.00 $2.64 $7.88
Moderate Even Azle $99.91 $83.26 $83.26 $24.56 $2,030.09 $1.08 $0.83 $5.29 $2,328.26
Moderate Even Motoko $99.91 $83.26 $83.26 $24.56 $4.54 $1.08 $0.83 $5.29 $302.72
Moderate Even Rust $99.91 $83.26 $83.26 $24.56 $439.23 $1.08 $0.83 $5.29 $737.41
Moderate Query Heavy Azle $50.45 $83.26 $0.83 $0.25 $20.30 $1.08 $0.83 $5.29 $162.29
Moderate Query Heavy Motoko $50.45 $83.26 $0.83 $0.25 $0.05 $1.08 $0.83 $5.29 $142.03
Moderate Query Heavy Rust $50.45 $83.26 $0.83 $0.25 $4.39 $1.08 $0.83 $5.29 $146.38
Moderate Update Heavy Azle $50.45 $0.83 $83.26 $24.56 $2,030.09 $1.08 $0.83 $5.29 $2,196.39
Moderate Update Heavy Motoko $50.45 $0.83 $83.26 $24.56 $4.54 $1.08 $0.83 $5.29 $170.85
Moderate Update Heavy Rust $50.45 $0.83 $83.26 $24.56 $439.23 $1.08 $0.83 $5.29 $605.53
Heavy Even Azle $9,990.60 $83,255.04 $83,255.04 $2,456.02 $203,008.59 $108.23 $832.55 $10.57 $382,916.65
Heavy Even Motoko $9,990.60 $83,255.04 $83,255.04 $2,456.02 $454.37 $108.23 $832.55 $10.57 $180,362.43
Heavy Even Rust $9,990.60 $83,255.04 $83,255.04 $2,456.02 $43,923.16 $108.23 $832.55 $10.57 $223,831.22
Heavy Query Heavy Azle $5,045.26 $83,255.04 $832.55 $24.56 $2,030.09 $108.23 $832.55 $10.57 $92,138.85
Heavy Query Heavy Motoko $5,045.26 $83,255.04 $832.55 $24.56 $4.54 $108.23 $832.55 $10.57 $90,113.31
Heavy Query Heavy Rust $5,045.26 $83,255.04 $832.55 $24.56 $439.23 $108.23 $832.55 $10.57 $90,547.99
Heavy Update Heavy Azle $5,045.26 $832.55 $83,255.04 $2,456.02 $203,008.59 $108.23 $832.55 $10.57 $295,548.81
Heavy Update Heavy Motoko $5,045.26 $832.55 $83,255.04 $2,456.02 $454.37 $108.23 $832.55 $10.57 $92,994.59
Heavy Update Heavy Rust $5,045.26 $832.55 $83,255.04 $2,456.02 $43,923.16 $108.23 $832.55 $10.57 $136,463.38
Median
USD Cost Estimates Per Year

These estimates use the median Wasm instructions per function call including the function prelude.

The Wasm instruction counts used are:

  • Azle: 2_387_322
  • Motoko: 5_169
  • Rust: 44_074
Cycle Costs Table

Cycle costs taken from here.

Compute Percent Allocated Per Second Update Message Execution Ten Update Instructions Execution Xnet Call Xnet Byte Transmission Ingress Message Reception Ingress Byte Reception GB Storage Per Second
100_000 590_000 4 260_000 1_000 1_200_000 2_000 127_000
Application Scenarios
Usage Query/Update Heaviness Ingress Bytes Per Query Message Ingress Bytes Per Update Message GB Storage Query Messages Per Second Update Messages Per Second Xnet Calls Per Second Xnet Call Bytes
Light Even 100 100 0.5 0.01 0.01 0.001 20
Light Query Heavy 100 100 0.5 0.01 0.0001 0.001 20
Light Update Heavy 100 100 0.5 0.0001 0.01 0.001 20
Moderate Even 1_000 1_000 1 1 1 0.1 200
Moderate Query Heavy 1_000 1_000 1 1 0.01 0.1 200
Moderate Update Heavy 1_000 1_000 1 0.01 1 0.1 200
Heavy Even 10_000 10_000 2 100 100 10 2_000
Heavy Query Heavy 10_000 10_000 2 100 1 10 2_000
Heavy Update Heavy 10_000 10_000 2 1 100 10 2_000
Application USD Cost Estimates Per Year
Usage Query/Update Heaviness CDK Ingress Messages Ingress Bytes Query Messages Ingress Bytes Update Messages Update Messages Update Instructions Xnet Calls Xnet Byte Transmission GB Storage Total Cost
Light Even Azle $1.00 $0.08 $0.08 $0.25 $0.40 $0.01 $0.00 $2.64 $4.46
Light Even Motoko $1.00 $0.08 $0.08 $0.25 $0.00 $0.01 $0.00 $2.64 $4.07
Light Even Rust $1.00 $0.08 $0.08 $0.25 $0.01 $0.01 $0.00 $2.64 $4.07
Light Query Heavy Azle $0.50 $0.08 $0.00 $0.00 $0.00 $0.01 $0.00 $2.64 $3.25
Light Query Heavy Motoko $0.50 $0.08 $0.00 $0.00 $0.00 $0.01 $0.00 $2.64 $3.25
Light Query Heavy Rust $0.50 $0.08 $0.00 $0.00 $0.00 $0.01 $0.00 $2.64 $3.25
Light Update Heavy Azle $0.50 $0.00 $0.08 $0.25 $0.40 $0.01 $0.00 $2.64 $3.89
Light Update Heavy Motoko $0.50 $0.00 $0.08 $0.25 $0.00 $0.01 $0.00 $2.64 $3.49
Light Update Heavy Rust $0.50 $0.00 $0.08 $0.25 $0.01 $0.01 $0.00 $2.64 $3.50
Moderate Even Azle $99.91 $83.26 $83.26 $24.56 $39.75 $1.08 $0.83 $5.29 $337.93
Moderate Even Motoko $99.91 $83.26 $83.26 $24.56 $0.09 $1.08 $0.83 $5.29 $298.26
Moderate Even Rust $99.91 $83.26 $83.26 $24.56 $0.73 $1.08 $0.83 $5.29 $298.91
Moderate Query Heavy Azle $50.45 $83.26 $0.83 $0.25 $0.40 $1.08 $0.83 $5.29 $142.38
Moderate Query Heavy Motoko $50.45 $83.26 $0.83 $0.25 $0.00 $1.08 $0.83 $5.29 $141.99
Moderate Query Heavy Rust $50.45 $83.26 $0.83 $0.25 $0.01 $1.08 $0.83 $5.29 $141.99
Moderate Update Heavy Azle $50.45 $0.83 $83.26 $24.56 $39.75 $1.08 $0.83 $5.29 $206.05
Moderate Update Heavy Motoko $50.45 $0.83 $83.26 $24.56 $0.09 $1.08 $0.83 $5.29 $166.39
Moderate Update Heavy Rust $50.45 $0.83 $83.26 $24.56 $0.73 $1.08 $0.83 $5.29 $167.04
Heavy Even Azle $9,990.60 $83,255.04 $83,255.04 $2,456.02 $3,975.13 $108.23 $832.55 $10.57 $183,883.20
Heavy Even Motoko $9,990.60 $83,255.04 $83,255.04 $2,456.02 $8.61 $108.23 $832.55 $10.57 $179,916.67
Heavy Even Rust $9,990.60 $83,255.04 $83,255.04 $2,456.02 $73.39 $108.23 $832.55 $10.57 $179,981.45
Heavy Query Heavy Azle $5,045.26 $83,255.04 $832.55 $24.56 $39.75 $108.23 $832.55 $10.57 $90,148.51
Heavy Query Heavy Motoko $5,045.26 $83,255.04 $832.55 $24.56 $0.09 $108.23 $832.55 $10.57 $90,108.85
Heavy Query Heavy Rust $5,045.26 $83,255.04 $832.55 $24.56 $0.73 $108.23 $832.55 $10.57 $90,109.50
Heavy Update Heavy Azle $5,045.26 $832.55 $83,255.04 $2,456.02 $3,975.13 $108.23 $832.55 $10.57 $96,515.36
Heavy Update Heavy Motoko $5,045.26 $832.55 $83,255.04 $2,456.02 $8.61 $108.23 $832.55 $10.57 $92,548.83
Heavy Update Heavy Rust $5,045.26 $832.55 $83,255.04 $2,456.02 $73.39 $108.23 $832.55 $10.57 $92,613.61
20 Likes
2

Great work! I’m not sure how to go about getting the garbage collection in there for motoko, but I’m guessing it will make it a bit more expensive than rust(not sure though).

How does gc work with azel? I’m guessing the engine is written in rust, so you don’t have to worry about it at that level, but does the engine itself need to garbage collect? And if so, when during the lifecycle does it do it?

2 Likes
3

Azle uses the Boa JS engine which is written in Rust. Boa itself as far as I know currently has a gc, this is the only information I really have on it: How to deal with Garbage Collection · Issue #363 · boa-dev/boa · GitHub

Seems there may be plans to remove the gc in the future. But for now I would assume gc is definitely running within the Boa context. The question is when it runs, and I’m not exactly sure. One comment in that thread I linked to said that about 4% of all Boa instructions were due to garbage collection, so possibly it’s a continuous cost, and then every once in a while it will need to do a larger loop through all garbage collected objects.

I haven’t dug into the Boa gc story too much yet, so I can’t really give more details.

2 Likes
4

This is great! Thank you!

I haven’t looked at the code but, for Motoko, your perf counter measurements are likely to be inaccurate because they won’t account for any gc costs, since GC happens after any computation (including your perf counter reads).

I’ve recently extended Motoko to give you access to both computation and gc costs of the last message, which might be useful to you once released.

3 Likes
5

Thank you so much for this.

Just a friendly suggestion. Having corresponding charts that are color coded by language and show the data for the 3 languages side by side for every parameter might make this data highly readable. Right now it’s a bit difficult to read given it’s in a tabular format

2 Likes
6

That would be great, we decided not to do that yet because of how much time we had already spent on it. We just wanted to get the raw data out, but that’s a natural next step that multiple people have suggested.

1 Like
7

Awesome, how will that work with the performance counter? Just hoping it will be automatic and not require any code changes

8

I think it will require some code changes. You’ll need to write the test as the body of a message and read the metrics after executing that message. It also won’t be possible, as it stands, to attribute instructions to serialization/deserialization, separately from compute, if you are interested in those numbers too.

9

azle is very good, allowing more developers to develop dfinity contracts using more familiar languages and richer class libraries, thank the author for his hard work. However, according to the above report, it is found that in many cases, the operating cost of azle is much higher than that of Motoko. This may be a serious problem in the case of a large number of application users. May I ask you to optimize and reduce the operation cost in the subsequent versions of azle cost?

10

Just in case anyone needs to know this, the numbers I shared in this thread are very very out of date now. Azle is much more performant now.

1 Like