TL;DR
- We provide an analysis showing that the current maximum burn-rate capacity of a 13-node subnet on the Internet Computer is 7’800T cycles per month. This is equivalent to 600T cycles per node per month.
- We show that the burn-rate capacity could be increased by a factor of 4 by the end of 2023 and by a factor of 20 by the end of 2026 via performance optimizations and enhancements in the cycle pricing framework.
- The maximum burn-rate capacity is a relevant metric for assessing the economic viability of the Internet Computer when assessed in conjunction with node provider rewards. We show that the above estimates, when assuming an average utilization of 30%, would lead to a surplus of cycles burned of 2100T cycles per node per month by the end of 2026.
Background and goal of the analysis
This article estimates the maximum burn-rate capacity of nodes on the Internet Computer (IC) in terms of how many cycles can be burned per month. The estimation covers both the current capacity and the future capacity, taking into account planned performance optimizations and enhancements in the cycle pricing framework.
For concreteness, the analysis focuses on a node within a 13-node subnet. Given that cycle pricing scales (mainly) linearly with the number of nodes, the results also apply to subnets with more or fewer nodes.
This analysis is a building block for further tokenomics assessments which will be provided in subsequent articles.
Current node capacity
We estimate the current node capacity in two ways: analytical and empirical. First, we provide an analytical assessment focusing on how many instructions, update & ingress messages a subnet can process in a given time period. Combining this with the cycle price of such operations yields an estimate for the cycle burn capacity. Second, we analyze the observed burn rate of heavily used subnet on the IC.
Scope of the analysis
In the analysis we focus on the capacity to process update & ingress messages and execute instructions because these are most relevant for the burn-rate capacity. These items correspond to approximately 80% of the current total cycle consumption of the IC. More precisely we include in the analysis
- Execution of update messages,
- Execution of instructions,
- Ingress message reception fees,
- Ingress byte reception fees,
- Same-subnet inter-canister call fees.
Out of scope of the analysis are operations as listed below. Please note that this is a conservative approach because including these additional aspects in the analysis would lead to higher estimates.
- Canister creations,
- Storage,
- Cross-subnet call fees,
- Cross-subnet byte fees,
- Same-subnet inter-canister byte transmission fees.
Analytical assessment
We consider several scenarios for estimating the monthly burn rate capacity of a node analytically.
Scenario 1: Maximum number of instructions
A subnet on the IC is currently running on 4 threads, each of which is capable of processing 2B instructions per second. Processing one instruction cost 0.4 cycles, as per the current IC pricing scheme. Thus a subnet, which is assumed to process the maximum number of instructions, burns 3.2B cycles per second. Dividing this burn-rate by the number of 13 nodes, yields a burn rate of 0.246B cycles per node per second. This is equivalent to 647T cycles per node per month.
Scenario 2: Maximum number of update messages
In this scenario we assume that a subnet processes the maximum number of update messages. Each of the 4 threads of a subnet can process approximately 1K update messages per second. Processing one update message costs 590K cycles. Thus a subnet that is assumed to process the maximum number of updates messages burns 2.36B cycles per second. Dividing this burn-rate by the number of 13 nodes, yields an average burn rate of 0.182B cycles per node per second. This is equivalent to 477T cycles per node per month.
Scenario 3: Maximum number of update messages in combination with ingress message reception
This scenario is a variation of the second scenario, combining update messages triggered by inter-canister calls and update messages triggered by users (called ingress messages). A subnet can process 1K ingress message receptions per second, and hence we assume that a subnet processes 3K updates messages triggered by inter-canister calls and 1K update ingress messages. One inter-canister call costs 0.26M cycles and one ingress message reception costs 1.2M cycles. Thus, in this scenario, a subnet would burn 2.36B cycles per second for the execution of 4K update messages and an additional 0.78B cycles for 3K inter-canister calls and 1.2B cycles for 1K ingress message reception per second. This sums up to 4.24B cycles per second. Dividing this burn-rate by the number of 13 nodes, yields a burn rate of 0.334B cycles per node per second. This is equivalent to 878T cycles per node per month.
Scenario 4: Maximum ingress byte reception
As opposed to Scenario 3, the focus of this scenario is not on processing the maximum of (small) ingress messages, but rather on processing a few big ingress messages. A subnet can process 2MB of ingress byte reception per second. Processing one ingress byte reception costs 2’000 cycles. Thus, in this scenario, a subnet would burn 4B cycles for ingress message reception per second. Dividing this burn-rate by the number of 13 nodes, yields a burn rate of 0.308B cycles per node per second. This is equivalent to 809T cycles per node per month.
Empirical assessment
In order to validate the above analytical assessments we analyzed the peak load observed for the heavily used subnet o3ow2 from Dec 06 to Dec 09 ‘22. During this time period of 3 days, this subnet burned 643T cycles out of which 12T cycles were due to canister creation and the remainder of 631T cycles due to message processing. Extrapolating this number to a full month and dividing by the number of nodes, provides an empirical estimate of the maximum burn-rate capacity of 485 T = 631 * 10/13T cycles per node per month.
Based on the above analysis we estimate that the current maximum burn-rate capacity of a node is approximately 600T cycles per month which is slightly below the average of the considered analytical scenarios. This estimate is aligned with the empirical estimate of 485T cycles.
Capacity improvements
In this section we provide an overview of suggestions on how to improve the execution & burn-rate capacity of subnets. Some of these ideas are already very concrete and planned to be implemented in 2023.
Increasing the number of execution threads
Currently a subnet has 4 update execution threads and under-utilizes the nodes, which have 64 cores. It is planned to increase the number of execution threads to 8 in the first half of 2023, thus improving the capacity by a factor of 2. A similar increase is planned for the number of query execution threads, which will be increased from 2 to 4 in the first half of 2023.
Optimizing the memory system
Since the launch of the Internet Computer, the message execution time has already decreased by a factor of 10x, by managing the memory throughput. There are additional ways how the memory system could be improved
- Wasm-native stable memory improves the performance of stable reads and writes. For certain workloads it would improve the performance of stable reads and writes by a factor of 2. Work on this feature is ongoing.
- Optimizing the implementation of orthogonal persistence. Currently, the implementation uses a signal handler and page protection. We think we can leverage userfaultfd or a different representation of the memory file to improve performance.
Over the course of 4 years, we believe that the burn-rate capacity of nodes can be improved by a factor of 10 which is similar to the gains observed since genesis in May 2021. Out of the overall target of 10x, we expect an improvement of at least 2x in the first half of 2023 by increasing the number of threads for update and query execution. The remaining improvement will be mainly driven by further optimizations to the memory system.
Enhancements of the cycle pricing framework
In this section we collect suggestions on how the cycle pricing framework could be enhanced and use it to estimate the impact on the burn rate. The guiding principle of the collected ideas is to charge operations in proportion to the resource usage. This seems natural from a fairness perspective and provides developers an incentive to optimize their applications for computational efficiency.
Charging for additional operations leads to a higher actual burn rate and also to a higher burn-rate capacity.
It is understood that the individual ideas for enhancing the cycle pricing framework would require detailed discussions & syndications before a consensus is reached on how to proceed (or not to proceed) with them.
Fair charging for instructions
Currently all WebAssembly instructions have the same cost in cycles. However, there are cheap instructions like adding numbers and expensive instructions like accessing memory. It would be desirable to charge for each WebAssembly instruction proportionally to its real cost. However to estimate the impact of such a scheme, further work on the design would be required.
Charging for query calls
At present, the Internet Computer does not charge for the execution of non-replicated query calls, as opposed to replicated update calls.
Given that query calls are non-replicated, the calculation cost should be an order of magnitude lower, namely 1/13 given that 13 is the number of nodes in a subnet, compared to the charge of updates calls. For the same reason, a subnet should be able to process 13 times more query calls than update calls.
Thus, balancing these factors, we expect that introducing charging for query calls could lead to an additional burn rate equal to the current burn-rate of update calls. This might be partially offset by mitigating actions by developers, as the new pricing scheme would provide an incentive for developers to optimize their queries.
The above estimate seems also in line with the current actual relationship of instructions triggered by query calls in comparison to update calls. As per Jan 27 ‘23, we observed the following:
- We see twice as many query calls (3K/sec) as update calls (1.6K/sec).
- When comparing the average number of Wasm instructions executed in a message in an execution round, we observe 500M/sec for query calls compared to 66M/sec for update calls.
- Thus the number of instructions triggered by query calls is an order of magnitude higher (approximately 14x) compared to update calls.
Conclusion
We estimate that the current maximum burn-rate capacity of a node in a 13-node subnet is approximately 600T cycles per month.
Via technical improvements over the course of 4 years, we believe that the burn-rate capacity of nodes can be increased by a factor of 10, which is similar to the gains observed since genesis in May 2021. Out of the factor of 10, we expect at least a factor of 2 to be realized in 2023.
In addition, via enhancements of the cycle pricing framework, we believe that the burn-rate could be increased by a further factor of 2, which could be realized in 2023.
In total this would increase the burn-rate capacity from 600T to 2400T cycles per node per month in 2023 and to 12000T cycles per node per month over the course of 4 years.
Based on the draft remuneration model for the second generation of nodes, we assume an average cost per node, i.e., node provider rewards from the NNS to node providers, of 1500 XDR which corresponds to 1500T cycles per month.
Combining the above projections, we conclude the following
- On subnets with an assumed average utilization of 30% we would reach a revenue (cycles paid by developers) of 3600T=12000T* 30% cycles per node.
- The average cost (rewards paid out to node providers) per node is 1500T cycles.
- Hence, the surplus of cycles burned, i.e. the income, per node would be 2100T = 3600T - 1500T cycles per month.
- Therefore, the operating margin per node, defined by income/revenue, would be 58%=2100/3600.
