Subnet storage capacity increase
The replica version 76fd768b increases the subnet storage capacity from 450GiB to 700GiB. This is the first milestone in a larger effort to push the subnet storage capacity into the terabytes.
Historically, the IC nodes had 3.2TB of NVMe space available at Genesis. Since a node may store multiple versions of the replicated state, the subnet capacity was conservatively limited to 450GiB. Besides that, more optimizations were needed in the state manager to scale beyond that limit.
Recently, the node providers in collaboration with DFINITY upgraded the nodes to increase the available NVMe space by 10x, from 3.2TB to 32TB. In parallel, DFINITY engineers have been working on performance optimizations in the state manager. All this work made it possible to increase the storage capacity to 700GiB. There are more potential state manager optimizations, which would allow further increases of the storage capacity, such as, for example, the rewrite of the storage layer to replace the XFS reflink operation during checkpointing.
Resource reservation mechanism
We took the newly added 250GiB as an opportunity to start tackling a long-standing problem with the storage charging mechanism. As you might know, currently canisters pay for storage dynamically every few minutes following the âpay-as-you-goâ model. Such a fine-grained payment is convenient for developers, but at the same time it doesnât handle spikey usage patterns well.
Consider a scenario where someone allocates the entire subnet storage for a few hours and pays only for those hours. The end result is that during those hours, the operation of other canisters on the same subnet might be disrupted as they might fail to allocate new storage. The problem here is that the cost of such a spikey usage is low due to the âpay-as-you-goâ model.
As a mitigation, the newly added 250GiB will be subject to a new resource reservation mechanism designed to encourage consistent, long-term use and discourage sudden, large spikes in storage usage. The mechanism is enabled only on application subnets, because verified application and system subnets do not need such protection since malicious actors cannot deploy canisters there.
The reservation mechanism works as follows:
- As long as the subnet remains under the previous limit of 450GiB, the storage payment remains the same as before, following the âpay-as-you-goâ model. This is the case for all subnets as of this writing.
- When the subnet grows above 450GiB, then the new reservation mechanism activates. Every time a canister allocates new storage bytes, the system sets aside some amount of cycles from the main balance of the canister. These reserved cycles will be used to cover future payments for the newly allocated bytes. The reserved cycles are not transferable and the amount of reserved cycles depends on how full the subnet is. For example, it may cover days, months, or even years of payments for the newly allocated bytes. It is important to note that the reservation mechanism applies only to the newly allocated bytes and does not apply to the storage already in use by the canister.
The goal of the reservation mechanism is to discourage the spiky usage pattern by making it more expensive while at the same time keeping costs the same for long-term users. As explained above, the reservation mechanism activates only above the previous subnet storage limit of 450GiB.
Details of the changes
- A new field named
reserved_cyclesis added to the canister state. The new field keeps track of the reserved cycles that were set aside from the main balance of the canister and were not spent yet. The new field will appear in the result ofcanister_statuscall to the IC management canister. - A new field name
reserved_cycles_limitis added to canister settings. It serves as an upper limit onreserved_cycles. Setting the limit to 0 is a way to disable the reservation mechanism for the canister. Such opted-out canisters would not be able to allocate from the newly added 250GiB, which means that these canisters will trap if they try to allocate storage when the subnet usage grows above 450GiB. In other words, such canisters will effectively operate under the previous subnet storage capacity of 450GiB. The default value of the newreserved_cycles_limitis 5T cycles. - Storage allocation operations such as
memory.grow,stable_grow,stable64_grow, settingmemory_allocationare adjusted to move cycles from the main balance toreserved_cycles. The amount of cycles depends on how full the subnet storage is: code. The amount of cycles per byte grows linearly with the subnet usage. It starts from 0 at 450GiB usage and goes to ~10 years worth of storage fees at 700GiB. These config parameters are up to discussion and may change after the community feedback. - The periodic charging for storage is adjusted to first burn cycles from
reserved_cyclesand only when that reaches 0, to start using the main balance. - The freezing threshold computation is also updated to take
reserved_cyclesinto account. This means that even if the main balance is below the freezing threshold, the canister may still be functional if it has enough reserved_cycles.
Next steps
Normally, changes in the protocol first go through the community discussion, then the NNS proposal, and then roll out to the mainnet. In this case, because of the possibility that this vulnerability of the protocol could be abused, we followed the approach for security fixes, where the fix is deployed first and then discussed later. Note that there are no changes for the first 450GiB that were available before. The new mechanism activates only above that threshold.
The plan:
- Implement and deploy the reservation mechanism on the mainnet [done].
- Implement the canister status and settings changes in dfx and agents [WIP]. A new version of dfx will be released soon.
- Discuss the reservation mechanism and other solutions [we are here now].
- Address issues raised by the community.
- Submit an NNS motion proposal.
- If the proposal is rejected, then we need to find and implement an alternative solution.
- If the proposal is accepted, then update the protocol specification.
What does it mean for you?
- If your canisters are on a verified application or system subnet, then you donât have to do anything because the reservation mechanism is disabled on such subnets.
- If you want to opt out, then you can set a new
reserved_cycles_limitcanister setting to 0 for all your canisters. In that case, the previous limit of 450GiB will apply to your canisters. - Discuss the problem and the reservation mechanism. Your feedback would be very much appreciated.
