How to calculate cycles balance not to raise "Couldn't send message"error

Developers Rust
1

Hello, I am a member of the Chainsight team.

I am seeing “Couldn’t send message” error sometimes in a canister that does inter-canister-call multiple times. To investigate the cause of this, we have created the following sample canister
ttps://github.com/hide-yoshi/canister_benchmarks

This is a sample of 10 canisters and makes inter canister calls from the first to the second, from the second to the third, … The inter canister call is made in the order from the first to the second, from the second to the third, …, from the ninth to the tenth canister.
The inter canister call is executed with the number of parallelism specified by the argument of the function call_server_update_parallel.
Also, inter canister calls are executed across subnets.

Checking through this, I found the following

  • Couldn’t send message error occurs when the number of parallel executions is increased
  • When Cycles are added to the canister and deposited, the error does not occur.

I found that if there are enough Cycles in the canister, the error does not occur.

So we decided to find out how many Cycles are enough.

The Cycles for an inter canister call is 5M, so if we were to make 100 requests, we would need 500M.

2.5T should be enough to call, but in practice, the call fails.

dev@dev-System-Product-Name:~/canister_benchmarks$ dfx canister status proxy1 --ic
WARN: The default identity is not stored securely. Do not use it to control a lot of cycles/ICP. Create a new identity with `dfx identity new` and use it in mainnet-facing commands with the `--identity` flag
Canister status call result for proxy1.
Status: Running
Controllers: br3vc-gyaaa-aaaao-a2qha-cai yuk46-o67cx-p3ddo-vzb27-o3cp4-owe4v-u4r3g-htj3u-4v2h6-sne4q-wqe
Memory allocation: 0
Compute allocation: 0
Freezing threshold: 2_592_000
Idle cycles burned per day: 19_750_229
Memory Size: Nat(1932656)
Balance: 2_596_519_566_784 Cycles
Reserved: 0 Cycles
Reserved cycles limit: 5_000_000_000_000 Cycles
Wasm memory limit: 3_221_225_472 Bytes
Module hash: 0xe51752d817d15b3a713af8509d1c9ea99bdbefac87558f7888b00a4daa0b93a9
Number of queries: 0
Instructions spent in queries: 0
Total query request payload size (bytes): 0
Total query response payload size (bytes): 0
Log visibility: controllers
dev@dev-System-Product-Name:~/canister_benchmarks$ dfx canister call proxy1 --ic call_server_update_parallel 100
WARN: The default identity is not stored securely. Do not use it to control a lot of cycles/ICP. Create a new identity with `dfx identity new` and use it in mainnet-facing commands with the `--identity` flag
(
  variant {
    Err = vec { "SysTransient: Couldn't send message"; "SysTransient: Couldn't send message"; "SysTransient: Couldn't send message"; "SysTransient: Couldn't send message"; "SysTransient: Couldn't send message"; "SysTransient: Couldn't send message"; "SysTransient: Couldn't send message"; "SysTransient: Couldn't send message"; "SysTransient: Couldn't send message"; "SysTransient: Couldn't send message"; "SysTransient: Couldn't send message"; "SysTransient: Couldn't send message"; "SysTransient: Couldn't send message"; "SysTransient: Couldn't send message"; "SysTransient: Couldn't send message"; "SysTransient: Couldn't send message"; "SysTransient: Couldn't send message"; "SysTransient: Couldn't send message"; "SysTransient: Couldn't send message"; "SysTransient: Couldn't send message"; "SysTransient: Couldn't send message"; "SysTransient: Couldn't send message"; "SysTransient: Couldn't send message"; "SysTransient: Couldn't send message"; "SysTransient: Couldn't send message"; "SysTransient: Couldn't send message"; "SysTransient: Couldn't send message"; "SysTransient: Couldn't send message"; "SysTransient: Couldn't send message"; "SysTransient: Couldn't send message"; "SysTransient: Couldn't send message"; "SysTransient: Couldn't send message"; "SysTransient: Couldn't send message"; "SysTransient: Couldn't send message"; "SysTransient: Couldn't send message"; "SysTransient: Couldn't send message"; "SysTransient: Couldn't send message"; "SysTransient: Couldn't send message"; "SysTransient: Couldn't send message"; "SysTransient: Couldn't send message"; "SysTransient: Couldn't send message";}
  },
)

In this example, the call succeeded after depositing an additional 5T Cycles.

dev@dev-System-Product-Name:~/canister_benchmarks$ dfx ledger --ic top-up proxy1 --amount 1
WARN: The default identity is not stored securely. Do not use it to control a lot of cycles/ICP. Create a new identity with `dfx identity new` and use it in mainnet-facing commands with the `--identity` flag
dfx canister staTransfer sent at block height 20243842
Using transfer at block height 20243842
tus --ic Canister was topped up with 5225900000000 cycles!

WARN: The default identity is not stored securely. Do not use it to control a lot of cycles/ICP. Create a new identity with `dfx identity new` and use it in mainnet-facing commands with the `--identity` flag
Canister status call result for proxy1.
Status: Running
Controllers: br3vc-gyaaa-aaaao-a2qha-cai yuk46-o67cx-p3ddo-vzb27-o3cp4-owe4v-u4r3g-htj3u-4v2h6-sne4q-wqe
Memory allocation: 0
Compute allocation: 0
Freezing threshold: 2_592_000
Idle cycles burned per day: 19_750_229
Memory Size: Nat(1932656)
Balance: 7_822_093_122_214 Cycles
Reserved: 0 Cycles
Reserved cycles limit: 5_000_000_000_000 Cycles
Wasm memory limit: 3_221_225_472 Bytes
Module hash: 0xe51752d817d15b3a713af8509d1c9ea99bdbefac87558f7888b00a4daa0b93a9
Number of queries: 0
Instructions spent in queries: 0
Total query request payload size (bytes): 0
Total query response payload size (bytes): 0
Log visibility: public

dev@dev-System-Product-Name:~/canister_benchmarks$ dfx canister call proxy1 --ic call_server_update_parallel 100
WARN: The default identity is not stored securely. Do not use it to control a lot of cycles/ICP. Create a new identity with `dfx identity new` and use it in mainnet-facing commands with the `--identity` flag
(variant { Ok = 33_373 : nat64 })

dev@dev-System-Product-Name:~/canister_benchmarks$ dfx canister status --ic proxy1
WARN: The default identity is not stored securely. Do not use it to control a lot of cycles/ICP. Create a new identity with `dfx identity new` and use it in mainnet-facing commands with the `--identity` flag
Canister status call result for proxy1.
Status: Running
Controllers: br3vc-gyaaa-aaaao-a2qha-cai yuk46-o67cx-p3ddo-vzb27-o3cp4-owe4v-u4r3g-htj3u-4v2h6-sne4q-wqe
Memory allocation: 0
Compute allocation: 0
Freezing threshold: 2_592_000
Idle cycles burned per day: 19_750_229
Memory Size: Nat(1932656)
Balance: 7_821_551_499_529 Cycles
Reserved: 0 Cycles
Reserved cycles limit: 5_000_000_000_000 Cycles
Wasm memory limit: 3_221_225_472 Bytes
Module hash: 0xe51752d817d15b3a713af8509d1c9ea99bdbefac87558f7888b00a4daa0b93a9
Number of queries: 0
Instructions spent in queries: 0
Total query request payload size (bytes): 0
Total query response payload size (bytes): 0
Log visibility: public

However, looking at the cycles balance after execution, there are enough cycles for execution even before deposit.
How do I calculate the cycles amount needed to make n inter canister calls?

2

A single inter-canister call cost consists of the following parts:

  1. The base fee for making the call which is 260K cycles.
  2. The cost for transmitting the request and response bytes. The cost per byte is 1K cycles.
  3. The cost for executing the response you will eventually receive back.

Because we cannot know the size of the response or how many cycles it’ll need during execution, we actually precharge for the maximum allowed and then refund accordingly when the response arrives.

The maximum response allowed is 2MiB. And the maximum execution allowed (with DTS) is 40B instructions. This means that the maximum cost for response execution is 5M cycles (base fee) + 40B cycles (at a price of 1 cycle per instruction executed).

The final formula is therefore:

inter-canister call cost = xnet_base_fee + request_size * xnet_byte_fee + max_response_size * xnet_byte_fee + execution_base_fee + max_instructions * instruction_fee = 260K + request_size * 1K + 2097152 * 1K + 5M + 40B*1 = 42102412000 + 1000 * request_size cycles

Note that in all the above the cycles costs are for a 13-node subnet which is the default application subnet size (they simply scale linearly for different subnet sizes).

We are additionally working on exposing a system API that canisters can call and figure out the cost of making an inter-canister call programmatically (clearly the above is not super easy to assemble and you’d need to ensure you keep your calculation up to date in case something changes on the platform level).

References:

  1. Cycles costs
  2. Resource limits
2 Likes