The above subnet information is illustrated below, followed by a node reference table:
Map Description
Red marker represents a removed node (transparent center for overlap visibility)
Green marker represents an added node
Blue marker represents an unchanged node
Highlighted patches represent the country the above nodes sit within (red if the country is removed, green if added, otherwise grey)
Light grey markers with yellow borders are examples of unassigned nodes that would be viable candidates for joining the subnet according to formal decentralisation coefficients (so this proposal can be viewed in the context of alternative solutions that are not being used)
Known Neurons to follow if you're too busy to keep on top of things like this
If you found this analysis helpful and would like to follow the vote of the LORIMER known neuron in the future, consider configuring LORIMER as a followee for the Subnet Management topic.
Other good neurons to follow:
Synapse (follows the LORIMER and CodeGov known neurons for Subnet Management, and is a generally well informed known neuron to follow on numerous other topics)
CodeGov (actively reviews and votes on Subnet Management proposals, and is well informed on numerous other technical topics)
WaterNeuron (the WaterNeuron DAO frequently discuss proposals like this in order to vote responsibly based on DAO consensus)
This proposal replaces a node in subnet uzr34: suty3, which appears as “Status: Degraded” in the IC dashboard, with another node from the same node provider and data centre thereby having no impact on Nakamoto coefficients or target topology parameters.
DFINITY will submit an NNS proposal today to reduce the notarization delay on the Internet Identity subnet, uzr34, similar to what has happened on other subnets in recent weeks (you can find all details in this forum thread).
After the successful rollout of the Application subnets, we propose a gradual rollout for the System subnets, starting with the Internet Identity subnet.
Thanks for this announcement @dsharifi, it seems to have aligned perfectly with my lunch break
Are you able to elaborate on the choice of subnet? The SNS subnet is technically an application subnet (albeit a special one). It has 34 nodes. I’d expect this to be a safer choice for starting the production changes on the large subnets (or maybe the fiduciary subnet). If something unexpected goes wrong in production, the II subnet would probably have one of the highest blast radiuses wouldn’t it?
At the moment this change has only been deployed to 13 node subnets (all of them).
Voted to adopt proposal 133307. The subnet id and the delay are correct. I don’t think that having larger number of nodes per subnet would be any issue with this, there are other limits in place that protect the subnet.
Larger subnets take longer to disseminate artifacts. That’s why this subnet currently has a delay of 1000ms instead of 600ms.
@dsharifi is setting this subnet to 300ms intentional? Could you elaborate? Perhaps this relates to the optimisation in last week’s IC OS proposal making the delay adaptive based on network conditions?
You mean the dynamic delay ? Also would you have been more comfortable with a reduction to only 600 ? I guess that we will have to wait for official answer.
Also I see from using the ic-admin tool that the current notarisation delay is 1000ms, not 600ms as the proposal says. I presume this might have been a typo but is there a case for lowering it to 600ms first? Is there some testing to help guide this decision?
Indeed, the current notarization delay is 1000ms on the Internet Identity subnet and all other System subnets. I forgot to change and update our proposal template summary description to 1000ms when submitting the proposal. So yes, the 600ms in the summary is an error/typo.
Yes, we have done extensive benchmarks with testnets with 40 and 31 nodes that indicate it is safe to lower the notarization delay down to 300ms. In the benchmarks we also simulate RTT, packet loss, and bandwidth to mimic the production topology based on our metrics. The benchmarks are the same ones we did with the Application testnets. The benchmarks involve:
Stress testing the subnet with a high load of Ingress messages, filling every execution round.
Kill nodes on the subnet for an extended duration, then restart the node to verify that it can state sync, catch up, and participate in block-making
Thanks for the extra explanation @dsharifi. I’m afraid I have to reject the proposal as it fundamentally misrepresents itself to voters, which needs to be a no no (even if it’s by accident) to avoid building up potentially dangerous precedent (and promoting bad voting culture).
Regarding the change itself (aside from the wording), could you clarify why the same delay is being used for subnets that are more than twice the size of smaller subnets using that delay? My understanding is that finalisation rates are dependent on the size of the subnet. Are you planning to reduce the delay for 13 node subnets even further?
Could you also comment on the choice of subnet? Is the II subnet considered lower risk than the SNS or Fiduciary subnets? This is the first time such a change is being rolled out to a large subnet (and the magnitude of the difference compared to the existing configuration is significantly greater).
I’ve voted to reject proposal 133307. As clarified by @dsharifi above (and thank you for explaining!) the current notarisation delay was mistakenly given as 1000ms in the proposal instead of 600ms. This is a key detail as it means that the magnitude of the change is greater than what voters may be given to understand. The description of the testing is very reassuring. However, from looking through recent Subnet Management proposals I’ve noticed that a number of nodes have had a sharp decrease in performance and have been listed for removal (from a subnet) following the decrease in notarisation delay for their respective subnets. Is this a valid concern and perhaps a greater risk for the larger subnets? I’m leaning towards favouring a smaller decrease at first, perhaps to 600ms, but I’m very open to being persuaded otherwise.
Voted to reject proposal 133307. Thanks for the thorough explanation on the proposal but I have to agree that even if it is just a simple mistake regarding the previous notarization delay mentioned in the proposal, it still might mislead people that might have different opinions otherwise as @timk11 and @Lorimer had. Also I think that providing more information specially when scaling this proposals to bigger and more relevant subnets should be done initially and I also would like to hear more on how the stress testing on this subnets work, for example do you simulate the distances between nodes that provide a similar behavior with the targeted subnet? Is there a way to verify the performance of those testnets?
