This topic is intended to capture Subnet Management activities over time for the snjp4 subnet, providing a place to ask questions and make observations about the management of this subnet.
1 Like
DFINITY will submit an NNS proposal today to reduce the notarization delay on the subnet, snjp4, similar to what has happened on other subnets in recent weeks (you can find all details in this forum thread).
2 Likes
Voted to adopt, as per comments here.
Voted to adopt proposal 133060. The initial_notary_delay_millis is set to 300 and the subnet_id is correct.
Voted to adopt proposal 134175, as the reasoning is sound and the description matches the payload. This proposal replaces 1 dead node, qnc5v, which appears on the IC dashboard as “Status: Offline”. The proposed change leaves the Nakamoto coefficients unchanged and the target topology parameters within the requirements.
1 Like
Voted to adopt proposal #134175.
The proposal replaces one dead node from Stockholm 1, SE qnc5v with the Offline status as of November 14th as per NPR dashboard with wzr5d from Marseille, FR without any impact on decentralization.
1 Like
Proposal 134175
TLDR: I’ve voted to adopt.
1 offline Swedish node replaced with a node that resides somewhere else.
- ip-api.com places it in The Netherlands
- api.ip2location.io places it in Germany
- domain WHOIS lookup places it in Germany
- IC records consider this node to be in France (note that this record is not kept up-to-date, and reflects the original location)
Worst case it’s in Germany, and the country nakamoto coefficient would therefore be reduced by this proposal, but it would still be within the IC Target Topology limits. @SvenF are you able to provide any clarity regarding the true location of this node?
Business rules check results before the membership change:
- Subnet should have 1 DFINITY-owned node(s) for subnet recovery, got 0
Interesting, so every subnet should have at least one node from the ‘DFINITY Operations SA’ node provider. I can certainly assert this when reviewing future proposals. Presumably any proposal that does not conform to this should be rejected, is that correct @Sat? Otherwise there’d by no way for DFINITY to apply a CUP to recovery the subnet?
Decentralisation Stats
Subnet node distance stats (distance between any 2 nodes in the subnet) →
| Smallest Distance | Average Distance | Largest Distance | |
|---|---|---|---|
| EXISTING | 191.173 km | 8877.458 km | 18480.492 km |
| PROPOSED | 191.173 km | 8868.804 km (-0.1%) | 18480.492 km |
Subnet characteristic counts →
| Continents | Countries | Data Centers | Owners | Node Providers | Node Operator | |
|---|---|---|---|---|---|---|
| EXISTING | 5 | 13 | 13 | 13 | 13 | 13 |
| PROPOSED | 5 | 12 (-8.3%) | 13 | 13 | 13 | 13 |
This proposal slightly reduces decentralisation in terms of jurisdiction diversity.
Largest number of nodes with the same characteristic (e.g. continent, country, data center, etc.) →
| Continent | Country | Data Center | Owner | Node Provider | Node Operator | |
|---|---|---|---|---|---|---|
| EXISTING | 4 | 1 | 1 | 1 | 1 | 1 |
| PROPOSED | 4 | 2 (+100%) | 1 | 1 | 1 | 1 |
See here for acceptable limits → Motion 132136
The above subnet information is illustrated below, followed by a node reference table:
Map Description
- Red marker represents a removed node (transparent center for overlap visibility)
- Green marker represents an added node
- Blue marker represents an unchanged node
- Highlighted patches represent the country the above nodes sit within (red if the country is removed, green if added, otherwise grey)
- Light grey markers with yellow borders are examples of unassigned nodes that would be viable candidates for joining the subnet according to formal decentralisation coefficients (so this proposal can be viewed in the context of alternative solutions that are not being used)
Table
Known Neurons to follow if you're too busy to keep on top of things like this
If you found this analysis helpful and would like to follow the vote of the LORIMER known neuron in the future, consider configuring LORIMER as a followee for the Subnet Management topic.
Another good neuron to follow is Synapse (follows the LORIMER and CodeGov known neurons for Subnet Management, and is a generally well informed known neuron to follow on numerous other topics)
Correct. However, a DFINITY node can be added to the subnet during the subnet recovery process. So this is not extremely critical except for the nns subnet - which is why we have 3 DFINITY nodes in the nns subnet, one should be working.
To answer a potential question - the reason previous subnet configuration had zero DFINITY nodes from the decentralization code point of view is because the DFINITY node was offline.
2 Likes
Cool, thanks Sat. It sounds like there needs to be a part of the IC Target Topology that mentions this. At the moment it states that a subnet like the NNS should have no more than 1 node belonging to the same node provider (which seems to contradict that business rule).
Presumably DFINITY node providers should be identified by checking if there is ‘DFINITY’ somewhere in the name? These appear to be the current DFINITY nodes in the NNS:
- Node Machine: fd5e4-a2xzl-lxu7m-kjvn6-2arnt-jghro-rdrgx-zvvkd-j2hza-pbwl4-5qe - ICP Dashboard (DFINITY USA Research LLC)
- Node Machine: tg4ec-b2g4p-h42kd-k7zvb-6rls2-i2q7l-gtr4h-ymr6v-rrez4-w6fao-oqe - ICP Dashboard (DFINITY Operations SA)
- Node Machine: 6hkcx-vz4jv-4n33r-ywdvs-sefaa-tb2on-rac6s-4csut-iyahu-zmct2-5qe - ICP Dashboard (DFINITY Operations SA)
3 Likes
Voted to adopt proposal 134175. The proposal replaces one nodes from subnet snjp4:
Removed Node: qnc5v, Dashboard Status Offline
Added Node: wzr5d.
The proposal was verified using the DRE tool to verify the metrics stated. No improvement to decentralization was made.
1 Like
Thanks @Lorimer for pointing this out, yes this should definitely be added to the target topology description.
2 Likes
This node should be in Marseille. I will check with the PfOps team who is managing the Dfinity nodes.
2 Likes
I checked with the PfOPs team. If you query the RADb registry with network routing information for the 2a0b:21c0:b002:2:6801:b6ff:fe37:2a67, then the RADB entry for the prefix confirms that it belongs to AS21859 (Zenlayer) and is described as ZL-IDC-MRS1-W29565. The “MRS” in the description likely refers to Marseille.
You will see the following:
route6: 2a0b:21c0:b000::/36
origin: AS21859
descr: ZL-IDC-MRS1-W29565
mnt-by: MAINT-AS21859
changed: ipadmin@zenlayer.com 20200312 #13:03:14Z
source: RADB
last-modified: 2023-11-13T16:00:10Z
rpki-ov-state: valid
However, the conclusion from our previous discussion seems to be right in that IPv6 geolocation services may lag in reflecting this. The address 2a0b:21c0:b000::/36 may be advertised across multiple locations (e.g., Marseille, Frankfurt, Amsterdam), and geolocation services might generalize the entire range to a central location.
The monitoring tool with triangulation that is being develop can help, although it will not be perfect as a network provider may for example performance maintenance activities on certain network connections which will change the routing between two locations. Therefore, another approach that is being explored is to do (paper and physical) audits on node machines in DCs; this is being discussed in the Technical Working Group for Node Providers.
2 Likes
Thanks @SvenF. An inaccuracy can mean the difference between a proposal being passable or not. This could also mean some undesirable indeterminism. The subnet could appear to meet the IC Target Topology one second, and not the next. I imagine physical audits wouldn’t be able to be used as a reactive measure (but something scheduled and few and far between).
I suppose the most important thing is the history of the nodes apparent location. If this is tracked minute by minute, or hour by hour, then transient blips due to the network conditions could be more easily established (and ignored).
Maybe part of the node rewards monitoring should be responsible for logging the apparent location of each node over time. If it deviates too frequently and/or for too long, then it becomes a problem (whereas transient discrepancies could be ignored)?
Proposal 134258
TLDR: Looks good, I’ll vote to adopt this proposal. The DFINITY-controlled node in this subnet is down, and is being replaced with an unassigned DFINITY-controlled node. This means that after this proposal executes, there should be 1 functional DFINITY node in this subnet, which is a requirement for DFINITY to be able to action subnet recovery steps in disaster scenarios.
Decentralisation metrics remain unchanged by this proposal.
Motivation:
- replacing dead node wzr5d-7qa2x-tduot-6mkkq-ou4sc-5ktje-4hxqa-a75d5-gunkc-3xhyt-6qe
Decentralisation Stats
Subnet node distance stats (distance between any 2 nodes in the subnet) →
| Smallest Distance | Average Distance | Largest Distance | |
|---|---|---|---|
| EXISTING | 191.173 km | 8868.804 km | 18480.492 km |
| PROPOSED | 191.173 km | 8868.804 km | 18480.492 km |
Subnet characteristic counts →
| Continents | Countries | Data Centers | Owners | Node Providers | Node Operator | |
|---|---|---|---|---|---|---|
| EXISTING | 5 | 12 | 13 | 13 | 13 | 13 |
| PROPOSED | 5 | 12 | 13 | 13 | 13 | 13 |
Largest number of nodes with the same characteristic (e.g. continent, country, data center, etc.) →
| Continent | Country | Data Center | Owner | Node Provider | Node Operator | |
|---|---|---|---|---|---|---|
| EXISTING | 4 | 2 | 1 | 1 | 1 | 1 |
| PROPOSED | 4 | 2 | 1 | 1 | 1 | 1 |
See here for acceptable limits → Motion 132136
The above subnet information is illustrated below, followed by a node reference table:
Map Description
- Red marker represents a removed node (transparent center for overlap visibility)
- Green marker represents an added node
- Blue marker represents an unchanged node
- Highlighted patches represent the country the above nodes sit within (red if the country is removed, green if added, otherwise grey)
- Light grey markers with yellow borders are examples of unassigned nodes that would be viable candidates for joining the subnet according to formal decentralisation coefficients (so this proposal can be viewed in the context of alternative solutions that are not being used)
Table
Known Neurons to follow if you're too busy to keep on top of things like this
If you found this analysis helpful and would like to follow the vote of the LORIMER known neuron in the future, consider configuring LORIMER as a followee for the Subnet Management topic.
Another good neuron to follow is Synapse (follows the LORIMER and CodeGov known neurons for Subnet Management, and is a generally well informed known neuron to follow on numerous other topics)
1 Like
Here is my vote for the last proposal.
Voted to adopt proposal 134258.
This proposal replaces node wzr5d which appears in the dashboard as “Status: Offline”. As shown in the proposal and verified using the DRE tool, decentralisation parameters are unchanged and remain within the requirements of the target topology.
1 Like
Voted to adopt proposal 134258. The proposal replaces a dead DFINITY operated node with a healthy one on subnet snjp4:
Removed Node: wzr5d, Dashboard Status Offline
Added Node: lrc24
The proposal was verified using the DRE tool to verify the metrics stated. There was no impact the the decentralization across all features.
1 Like
That’s a good idea and I will bring it into the discussion, not sure how complex it will be to make the logging for triangulation trustworthy by including it in the reward monitoring tool, but it is definitely something to consider.
1 Like