DFINITY voted REJECT on Proposal 126111, which intended to replace 4 nodes in subnet lhg73. The purpose of this post is to explain that vote.
Already late last week, two nodes in subnet lhg73 were unresponsive. The affected nodes were skzfd-vbz7c-5cbqo-6ejuw-nf6ks-c7doo-njtlq-gbcmw-smuwf-xes7h-4ae and wazkf-2xehg-y26ek-2uq64-pkrbh-6frmq-4q7lf-5gaya-44q4p-v5r5z-cqe. In addition, a third node tq3rq-mnjxc-3szli-4qntc-2nojp-telh7-kav7a-hjlyy-odlqr-lp2ie-pae showed flaky network behavior. The aim of proposal 126111 was to replace the above three nodes as well as a fourth node, a2e7m-evijx-geoc6-o5j6s-h45ye-f3hxj-w7dfi-25e2k-v5ylv-nf7ud-aqe. The purpose of this last replacement was to improve the decentralization parameters of the subnet.
When nodes of a subnet are swapped, the “new” nodes first synchronize the subnet state. During that time, they cannot contribute to the consensus protocol: they may not have the necessary data to check the validity of blocks or to propose a valid new block. Thus, until the “new” nodes have caught up, the subnet is supported exclusively by the “old” nodes that remain in the subnet.
In an application subnet with 13 nodes, up to 4 nodes can misbehave arbitrarily without affecting the security or liveness of the subnet. Indeed, the subnet was functioning just fine despite the issues at the above-listed 3 nodes. When swapping 4 nodes simultaneously, however, the network is supported exclusively by the remaining 9 nodes, and any additional (node or network) failure will cause a delay in processing ingress messages.
It is important to understand that even in case of such a failure, the only effect would be a degraded performance until the new nodes caught up with the subnet state; as soon as sufficiently many new nodes are available, the subnet would proceed automatically. Of course, in a subnet with > 300 GB of subnet state such as lhg73, synchronizing the state to 4 new nodes simultaneously also creates significant load on the network. The decision of DFINITY to reject Proposal 126111 and instead submit Proposal 126301 was made in order to keep a safety margin even against temporarily degraded subnet performance.
If someone were reviewing this proposal and trying to verify it is consistent with best practices such as the one you just explained, where do you recommend we go to find the information needed to make that evaluation. I can see that the dashboard displays the number of node machines that are in the subnet along with their location, data center, and node provider. However, how do we know best practice for the max number of nodes that should be replaced with any given proposal? Also, how do we know if the proposed nodes are an improvement to the decentralization parameters of the subnet? Since proposal 126301 has already been executed, I can’t tell if there was previously two nodes with similar features, but I suppose a goal may be max diversity of location, data center, and node provider. I’m not sure what else to look for.
Also, I can see that the overview for this node shows the number of node machines. There appears to be a downward trend and a sudden increase here at the end. How can we tell the timeline of that trend and how are the number of node machines determined? Based on your description, it seems this trend may be a few days of data. Clicking the Node Machines link takes you to a trend of all node machines in all subnets, not a larger graphic of the 13 node machines assigned to this subnet.
Thanks again for the explanation. It was interesting to learn about these criteria for evaluating suitability of a proposal.
@wpb I can provide the answer around the decentralization parameters. We recently created a public DRE repo GitHub - dfinity/dre: Decentralized Reliability Engineering in which we will publish and share with the community the tooling that we internally use for calculating decentralization and submitting these proposals. The tooling gives an output similar to this, prior to the proposal submission:
Going forward, my team plans to work with the wider community on improving the transparency and the decentralization of these operational activities.
Also, given that we use standard Nakamoto coefficients to calculate decentralization, I don’t see a reason that some 3rd party tooling couldn’t be set up to keep track of subnet decentralization. That could also be much simpler than the Rust tooling we use for proposal submission.
The subnet protocol makes the assumption n >= 3f + 1. Which means that for n = 13 as in this subnet, we have f <= 4 and thus at least 9 = 13 - 4 nodes are needed to keep the subnet alive. The analogous computation works for any subnet size.
