This topic is intended to capture Subnet Management activities over time for the io67a subnet, providing a place to ask questions and make observations about the management of this subnet.
At the time of creating this topic the current subnet configuration is as follows:
This proposal sets the notarisation delay of the subnet to 300ms, down from 600ms. The change will increase the block rate of the subnet, aimed to reduce latency of update calls.
600ms: used by all other subnets (13 nodes subnets)
Larger subnets naturally require larger delays. However @dsharifi announced earlier today plans to increase the block rate, thanks to numerous performance enhancements that have been implemented.
@dsharifi are you able to provide some information about why this subnet was chosen to roll out this new config? Presumably eventually all 13 node subnets will use this configuration? What about the larger subnets, will you be aiming to keep the same relative proportions by halving the notary delay to 500ms (or lower)? (I’m only asking because I’m aware that there’s currently a drive to minimise configuration differences between NNS and other subnets).
@Lorimer 300ms is too low for nodes on the other side of the world.
This node is in sc1 data centre, an hour north of Brisbane in Australia. I have been monitoring ping times from a server on AWS in Frankfurt to a node on the io67a subnet q3w37-sdo2u-z72qf-hpesy-rgqes-lzflk-aescx-c5ivv-qdbty-s6pgc-jae :
Thanks @Lerak. Nice idea to get an indication of network latency. It would be interesting to hear more from the networking team about how representative they consider this to be (@dsharifi).
My understanding is that the initial_notary_delay_millis is the minimum amount of time that all nodes need to wait to notarize (but they can wait longer, with notarisation falling back to other ranked blocks). Setting this somewhere close to the network latency that can be reasonably expected under ideal conditions may be intentional (I’m not sure). If this is set too low for all nodes to be able to disseminate artifacts in time, my understanding is that it’s more likely that multiple notarised blocks at the same height will occur (but the subnet will still be able to make progress, albeit on multiple chains that will need pruning by the finalisation process).
@Manu, are you able to provide some insight into the trade-offs involved in setting the initial_notary_delay_millis for optimal subnet performance? (and worst and best case scenarios that can be expected)
Given that artifacts are pushed with the new P2P implementation, a node only needs 1/2 RTT for its notarization share to reach a peer.
We have done extensive performance tests, where we have simulated the network topology of the io67a subnet by simulating RTTs, packet loss, and bandwidth to mimic production settings.
Of course, if we see any regressions we will propose adjusting the delays again.
Hi @Lerak! I guess you’re thinking about the risk of nodes with higher ping being unable to propose blocks. So in addition to what @dsharifi said, note that this initial notary delay is only the start of when notaries start signing the block from the highest priority block maker. Only after initial_notary_delay + unit_delay (which would be 300 ms initial_notary_delay if this proposal passes + 1000ms unit delay) do notaries fall back to supporting blocks from lower priority block makers.
We will keep an eye on the block making metrics to ensure that no problems are introduced.
Thanks for creating this new thread specifically for the io67a proposal!
Yes, the goal is indeed for all 13 node subnets to be configured to have an initial notary delay of 300ms.
We don’t have a specific value in place yet for the larger subnets. We are still doing extensive benchmarks to ensure that we find a safe value such that all nodes can make block proposals, and keep up in larger subnets. As we are doing a gradual rollout, we will not propose to adjust the larger subnets until we have adjusted all 13 node Application subnets with metrics to back up that everything works as expected.
@Manu Yes - I am concerned that it can affect the node’s trustworthy metrics stats. Thus when this node is chosen to be the blockmaker - will there now be an increased probability of timing out?
Theoretically yes, right now the node has to propose its block in 1600 ms, and with this proposal it reduced to 1300 ms, so it’s not a huge reduction. We will keep an eye on the trustworthy node metrics to see if we see any noticeable differences, and please let us know if you notice anything.
Voted to adopt proposal 134040, as the reasoning is sound and the proposal description matches the payload.
This proposal is intended to replace a dead node, ahekl, which node appears as “Status: Offline” on the dashboard. As seen in the proposal (which I verified using the DRE tool), the proposed change leaves the target topology within the requirements and improves decentralisation with respect to country.
TLDR: This proposal replaces an offline node in the US with an unassigned node in Latvia. I’ve adopted.
This is a great proposal. The subnet is currently violating the IC Target Topology, with 3 nodes in the same country (when the limit is 2). After this proposal executes there will be no more than 2 nodes in the same country.
Decentralisation Stats
Subnet node distance stats (distance between any 2 nodes in the subnet) →
Smallest Distance
Average Distance
Largest Distance
EXISTING
304.223 km
8920.217 km
16748.078 km
PROPOSED
304.223 km
8500.17 km (-4.7%)
16748.078 km
This proposal slightly reduces decentralisation, considered purely in terms of geographic distance (and therefore there’s a slight theoretical reduction in localised disaster resilience).
Subnet characteristic counts →
Continents
Countries
Data Centers
Owners
Node Providers
Node Operator
EXISTING
5
11
13
13
13
13
PROPOSED
5
12 (+8.3%)
13
13
13
13
This proposal slightly improves decentralisation in terms of jurisdiction diversity.
Largest number of nodes with the same characteristic (e.g. continent, country, data center, etc.) →
The above subnet information is illustrated below, followed by a node reference table:
Map Description
Red marker represents a removed node (transparent center for overlap visibility)
Green marker represents an added node
Blue marker represents an unchanged node
Highlighted patches represent the country the above nodes sit within (red if the country is removed, green if added, otherwise grey)
Light grey markers with yellow borders are examples of unassigned nodes that would be viable candidates for joining the subnet according to formal decentralisation coefficients (so this proposal can be viewed in the context of alternative solutions that are not being used)
Known Neurons to follow if you're too busy to keep on top of things like this
If you found this analysis helpful and would like to follow the vote of the LORIMER known neuron in the future, consider configuring LORIMER as a followee for the Subnet Management topic.
Another good neuron to follow is Synapse (follows the LORIMER and CodeGov known neurons for Subnet Management, and is a generally well informed known neuron to follow on numerous other topics)
Voted to adopt proposal 134040. The proposal replaces one nodes from subnet io67a:
Removed Node: ahekl, Dashboard Status Offline
Added Node: poyg5.
The proposal was verified using the DRE tool to verify the metrics stated and on top of replacing the dead node, the replacement improves decentralization on the country metric, reducing the number of US nodes from 2 to 1.
The proposal is correct in that it replaces offline dead node from Jacksonville, US ahekl , with Riga1, LV node poyg5 while improving decentralization.
Hello @Manu,
When you made this reply, we were somewhat sceptical about the impact of reducing the notarization delay on the blockmaker failure rate. Over the past three months, we monitored the blockmaker failure rates for the two subnets (eq6en and w4rem) with the highest median failure rates that experienced a reduction in notarization delay from 600ms to 300ms. Below are the results.
Please note that the transparent lines in the graph represent the actual data points, while the solid lines show the 7-day moving averages. From the graph, it appears that there has been a drastic increase in the block failure rate.
These findings lead me to ask you and @dsharifi the following questions:
Each time a blockmaker fails to produce a block and a new blockmaker is chosen, doesn’t this overall increase the network latency (considering the time taken by the first blockmaker to fail and the time taken by the new blockmaker to create the block)?
Is there a specific threshold where Dfinity would consider the failure rate unacceptably high?
Yes, block makers missing their opportunity to create a block and having to fall back adds latency. However, always having a slower block rate also increases latency. In my view, we should optimize for latency, because that’s what users actually notice, and we do see that latency has gone down with the increased block rate
This proposal replaces node 2xph2 which appears in the dashboard as “Status: Active”, for the purpose of making it available to other subnets in order to improve overall network topology. As shown in the proposal and verified using the DRE tool, decentralisation parameters are unchanged and remain within the requirements of the target topology.
I have raised question regarding this kind of proposals previously and adopted them firstly expecting better explanation in the future as can be seen in this two reviews, 134191 and 134192.
I’ve voted ‘yes’
