Subnet Management - 6pbhf (Application)

Voted to adopt proposal 133080.

This proposal is intended to replace a dead node, l2bzb. However, this node appears as “Status: Active” on the dashboard. It currently appears as “status: UP” using the decentralization tool, but appeared as “alert: IC_Networking_StateSyncLoop” and “status: DEGRADED” when previously checked at 06:21 UTC (today, 21 Sept). As noted here, this node has been repeatedly degrading and recovering. As seen in the proposal (which I verified using the DRE tool), the proposed change leaves the Nakamoto coefficients and target topology parameters unchanged and within the requirements.

4 Likes

Thanks for this proposal @Sat, and great question @timk11 regarding the status of that node flip-flopping. I’d been intending on setting up an app that frequently polls node state in order to provide an easy way of visualising node status history. I wasn’t actually aware that we already have this sort of report (and more) with the Node Provider Rewards dashboard that Sat has highlight. This is great! :star_struck:

I’d be interested to learn more about who maintains this dashboard (I see it’s controlled by principal - 32kwa-eibir-hat2o-kyhfc-u5msq-xr4w4-newoz-qpzxj-ohovv-u7zgt-qqe). @Sat can I ask who this controller is, or what team this principal represents?


The circumstances around this proposal are actually super interesting :eyes:

Here are the block rate metrics for a randomly selected node in this subnet (as an exemplar point of reference) - jj7si

image

Note the 100% success rate, and that the block rate significantly increased at the start of September, as a direct result of this prior proposal (reducing notarization delay) → Subnet Management - 6pbhf (Application) - Governance / NNS proposal discussions - Internet Computer Developer Forum (dfinity.org)


Here are the block rate metrics for the node that keeps falling into a degraded state

image

We can see that it was already failing to keep up 100% with the existing block rate (prior to September). Interestingly the performance of this node in absolute terms has actually improved since the start of September (on average). However, in relative terms (relative to the increased total number of block), this node’s performance now borders on being considered degraded.

It might be worth looking out for nodes like this when decreasing the notarization delay on critical system subnets (which I suspect will happen soon).

@Sat, out of interest, can I ask what the exact criteria are for when a node status is considered to switch from UP to DEGRADED? Presumably the proportion of failed blocks needs to cross some limit?


Proposal 133080

Now that the preamble is out of the way, the ‘degraded’ node in Korea is being replaced with an ‘up’ node in Israel :+1: I’ve voted to adopt

There’s a slight reduction in the average distance between nodes if this proposal passes, but that seems fine (particularly given that it’s not a metric directly considered by the IC target topology). All other decentralisation metrics appear unaltered by this proposal.

Decentralisation Stats

Subnet node distance stats (distance between any 2 nodes in the subnet) →

Smallest Distance Average Distance Largest Distance
EXISTING 117.012 km 6915.788 km 17244.636 km
PROPOSED 117.012 km 6591.406 km (-4.7%) 17244.636 km

This proposal slightly reduces decentralisation, considered purely in terms of geographic distance (and therefore there’s a slight theoretical reduction in localised disaster resilience). :-1:

Subnet characteristic counts →

Continents Countries Data Centers Owners Node Providers
EXISTING 4 13 13 13 13
PROPOSED 4 13 13 13 13

Largest number of nodes with the same characteristic (e.g. continent, country, data center, etc.) →

Continent Country Data Center Owner Node Provider
EXISTING 7 1 1 1 1
PROPOSED 7 1 1 1 1

See here for acceptable limits → Motion 132136

The above subnet information is illustrated below, followed by a node reference table:

Map Description
  • Red marker represents a removed node (transparent center for overlap visibility)
  • Green marker represents an added node
  • Blue marker represents an unchanged node
  • Highlighted patches represent the country the above nodes sit within (red if the country is removed, green if added, otherwise grey)
  • Light grey markers with yellow borders are examples of unassigned nodes that would be viable candidates for joining the subnet according to formal decentralisation coefficients (so this proposal can be viewed in the context of alternative solutions that are not being used)

Table
Continent Country Data Center Owner Node Provider Node Status
--- Asia Korea (the Republic of) Seoul 2 (kr2) Gasan Web3game l2bzb-vpnu6-jlsae-poddg-5hpnz-bx54x-3khc2-aazpn-bop4m-zforf-uqe DEGRADED
+++ Asia Israel Tel Aviv 1 (tv1) Interhost GeoNodes LLC vwsvq-sp3z5-kjy66-srr2f-esws7-mzchn-oeqdb-n3qvz-wrzni-4pcsy-lae UNASSIGNED
Oceania Australia Queensland 1 (sc1) NEXTDC ANYPOINT PTY LTD r4dbq-jplty-hxb2n-yx4pt-dm2vf-m73ro-sqgq7-onnzn-zenvp-dleaz-mae UP
Europe Belgium Antwerp (an1) Datacenter United Allusion mehhn-b5swd-urm2r-cltwk-c5tns-2s2bf-skz74-hv63o-3qdad-ubbfn-iae UP
Europe Germany Geneva 2 (ge2) SafeHost Archery Blockchain SCSp 4mm7j-dmeng-7ib4h-yvt3c-g5ed6-i5yar-rrc6w-rp7tk-ej3by-gilvv-kae UP
Europe Spain Madrid 1 (ma1) Ginernet Artem Horodyskyi jj7si-b7rs7-nq7sv-npame-aec3y-2anip-ol5s6-gxbh7-kjbg2-a5u6j-rae UP
Asia Georgia Tbilisi 1 (tb1) Cloud9 George Bassadone xqhoe-c5pck-wcytx-owvyr-mntbp-u22ct-nvcqi-bri5w-sjow2-rs4gp-sae UP
Europe Croatia Zagreb 1 (zg1) Anonstake Anonstake rzm7j-37ied-ynlvt-zma7n-r4bjg-wxg2d-kr4me-ss4yr-3p5se-n4sry-wqe UP
Asia Japan Tokyo 2 (ty2) Equinix Starbase w53hu-bdzuz-h7h75-weodb-getvj-rr766-m2rtb-bigdq-l62cj-7atxw-2ae UP
Europe Romania Bucharest (bu1) M247 Iancu Aurel vuizy-nfm5v-rapnc-rijer-hijfx-bvjrz-ccxdd-kte3v-awbnq-bdm6m-6qe UP
Asia Singapore Singapore (sg1) Telin OneSixtyTwo Digital Capital l4mrq-cmo2o-ydidi-v2zit-pemyc-itm4j-qw2u3-kwzso-yz5dv-geium-pqe UP
Europe Slovenia Maribor (mb1) Posita.si Fractal Labs AG efdju-ef2ce-a5jdn-obybl-x6ema-h5lwv-nc2sy-v4hvc-7nltm-aldtv-6ae UP
Europe Sweden Stockholm 1 (sh1) Digital Realty DFINITY Operations SA tgmtp-wy3f4-hqron-bnvc3-scclx-b7fgg-gdnc2-dwvks-dn6ao-gbqso-5ae UP
Americas United States of America (the) San Jose (sj1) INAP Shelburne Ventures, LLC 7v72g-sof5q-riabw-dzefk-7p74b-wxwzs-dgvbv-rlrxx-2jpjy-zli4s-cqe UP

Known Neurons to follow if you're too busy to keep on top of things like this

If you found this analysis helpful and would like to follow the vote of the LORIMER known neuron in the future, consider configuring LORIMER as a followee for the Subnet Management topic.

Other good neurons to follow:

  • Synapse (follows the LORIMER and CodeGov known neurons for Subnet Management, and is a generally well informed known neuron to follow on numerous other topics)

  • CodeGov (actively reviews and votes on Subnet Management proposals, and is well informed on numerous other technical topics)

  • WaterNeuron (the WaterNeuron DAO frequently discuss proposals like this in order to vote responsibly based on DAO consensus)

2 Likes

Voted to adopt proposal 133080.

Currently the status of the node l2bzb appears as Degraded and the proposal replaces this node on subnet 6pbhf witht the node vwsvq. Using the Dre tool, I verified that there were no changes on the Nakamoto Coeffiecients and since this node has been unreliable as seen in the images from the Node Provider Rewards of both nodes.

Node l2bzb:

Node vwsqv:

2 Likes

Voted to adopt proposal 133080 since Motivation and Node details are correct.

1 Like

Voted to adopt proposal 134273. The proposal seeks to remove a cordoned node from the subnet and specifies that the associated data centre is being offboarded “after 48 months”. Decentralisation parameters are unchanged. The necessary context is provided by this forum post and associated discussion. For future proposals of this type I recommend that the background context be included in the proposal text for ease of verification.

I’ve voted to reject proposal 134273. It makes claims that I see no clear way of verifying, and supporting statements regarding the ownership of the data centre are inconsistent with records held in the IC registry. The proposal was announced here.

Voted to adopt proposal 134273. This proposal is part of a sequence of steps to remove cordoned nodes from subnets as the associated data centeres are being offboarded after 48 months of their respective DC contracts that are still private and were signed up before the Genesis. There is a great and detailed explanation of this changes in this forum post and the forum thread it is in. In the wiki there is a series of Steps for Gen-1 Node onboarding after 48 months that need to be followed in order for the nodes to continue earning rewards which starts by making a forum post in the following thread. As we can verify no one as come forward with nodes from the DCs in this proposals so I don’t see any issues with the removal of this nodes.

This subnet was recently made private.

Voted to adopt proposal 134399.

This proposal replaces node l4mrq which appears in the dashboard as “Status: Offline”. As shown in the proposal, decentralisation parameters are unchanged and remain within the requirements of the target topology.

Proposal 134399

Vote: ADOPT

The proposal replaces one node from subnet 6pbhf:
Removed Node: l4mrq: Dashboard Status Offline
Added Node: rptzo
The proposal was verified using the DRE tool to verify the metrics stated which are not affected by this replacement.

I’ve voted to adopt proposal 134399. The node that was replaced is still degraded, and the decentralisation coefficients for the subnet were not modified by this proposal.

A new proposal with id 134541 has been submitted, please take a look.

Click here to open proposal details

Replace nodes in subnet 6pbhf

Motivation:
The following nodes in subnet 6pbhf have been cordoned and need to be removed from the subnet:

Decentralization Nakamoto coefficient changes for subnet 6pbhf-qzpdk-kuqbr-pklfa-5ehhf-jfjps-zsj6q-57nrl-kzhpd-mu7hc-vae:

    node_provider: 5.00 -> 5.00    (+0%)
      data_center: 5.00 -> 5.00    (+0%)
data_center_owner: 5.00 -> 5.00    (+0%)
             area: 5.00 -> 5.00    (+0%)
          country: 5.00 -> 5.00    (+0%)

Mean Nakamoto comparison: 5.00 → 5.00 (+0%)

Overall replacement impact: equal decentralization across all features

Details

Nodes removed:

  • mehhn-b5swd-urm2r-cltwk-c5tns-2s2bf-skz74-hv63o-3qdad-ubbfn-iae [health: healthy]
  • rptzo-ymdmg-3zp32-xdvov-efmon-lgmet-pnbxl-4qjgy-nba6j-pcw3k-yqe [health: healthy]

Nodes added:

  • afjfz-erkja-k7euy-yjjrk-tsvsw-hfad7-allt6-4lb3e-kscys-7uyge-7ae [health: healthy]
  • 2yzcy-lywoi-fpfpg-ghagw-3emh7-2lj3p-ritfh-6e542-poww3-rafkw-wae [health: healthy]
    node_provider                                                              data_center            data_center_owner            area                    country        
    -------------                                                              -----------            -----------------            ----                    -------        
    6nbcy-kprg6-ax3db-kh3cz-7jllk-oceyh-jznhs-riguq-fvk6z-6tsds-rqe       1    an1          1 -> 0    Anonstake               1    Bucuresti          1    AU            1
    6sq7t-knkul-fko6h-xzvnf-ktbvr-jhx7r-hapzr-kjlek-whugy-zt6ip-xqe       1    aw1               1    Cloud9                  1    Flanders      1 -> 0    BE       1 -> 0
    7a4u2-gevsy-5c5fs-hsgri-n2kdz-dxxwf-btcfp-jykro-l4y7c-7xky2-aqe       1    bu1               1    Cyxtera            0 -> 1    Geneva             1    CA       0 -> 1
    7at4h-nhtvt-a4s55-jigss-wr2ha-ysxkn-e6w7x-7ggnm-qd3d5-ry66r-cae  0 -> 1    ge2               1    Datacenter United  1 -> 0    Madrid             1    CH            1
    7ryes-jnj73-bsyu4-lo6h7-lbxk5-x4ien-lylws-5qwzl-hxd5f-xjh3w-mqe       1    ma1               1    Digital Realty          1    Maribor            1    ES            1
    bvcsg-3od6r-jnydw-eysln-aql7w-td5zn-ay5m6-sibd2-jzojt-anwag-mqe       1    mb1               1    Equinix                 1    Ontario       0 -> 1    GE            1
    diyay-s4rfq-xnx23-zczwi-nptra-5254n-e4zn6-p7tqe-vqhzr-sd4gd-bqe       1    sc1               1    Ginernet                1    Pennsylvania       1    HR            1
    fwnmn-zn7yt-5jaia-fkxlr-dzwyu-keguq-npfxq-mc72w-exeae-n5thj-oae       1    sg1          0 -> 1    Interhost               1    Queensland         1    IL            1
    i7dto-bgkj2-xo5dx-cyrb7-zkk5y-q46eh-gz6iq-qkgyc-w4qte-scgtb-6ae       1    sg3          1 -> 0    M247                    1    Singapore          1    JP            1
    kos24-5xact-6aror-uofg2-tnvt6-dq3bk-c2c5z-jtptt-jbqvc-lmegy-qae       1    sh1               1    NEXTDC                  1    Stockholm          1    RO            1
    rbn2y-6vfsb-gv35j-4cyvy-pzbdu-e5aum-jzjg6-5b4n5-vuguf-ycubq-zae  1 -> 0    tb1               1    Posita.si               1    Tbilisi            1    SE            1
    sixix-2nyqd-t2k2v-vlsyz-dssko-ls4hl-hyij4-y7mdp-ja6cj-nsmpf-yae       1    to2          0 -> 1    Racks Central      1 -> 0    Tel Aviv           1    SG            1
    vegae-c4chr-aetfj-7gzuh-c23sx-u2paz-vmvbn-bcage-pu7lu-mptnn-eqe       1    tv1               1    SafeHost                1    Tokyo              1    SI            1
    wdjjk-blh44-lxm74-ojj43-rvgf4-j5rie-nm6xs-xvnuv-j3ptn-25t4v-6ae       1    ty2               1    Telin              0 -> 1    Zagreb             1    US            1
                                                                               zg1               1    Tierpoint               1                                           
1 Like

Proposal 134541

TLDR: To be continued…

Decentralisation Stats

Subnet node distance stats (distance between any 2 nodes in the subnet) →

Smallest Distance Average Distance Largest Distance
EXISTING 117.012 km 6387.421 km 17244.636 km
PROPOSED 117.012 km 6954.635 km (+8.9%) 17244.636 km

This proposal slightly increases decentralisation, considered purely in terms of geographic distance (and therefore there’s a slight theoretical increase in localised disaster resilience). :+1:

Subnet characteristic counts →

Continents Countries Data Centers Owners Node Providers Node Operator
EXISTING 4 13 13 13 13 13
PROPOSED 4 13 13 13 13 13

Largest number of nodes with the same characteristic (e.g. continent, country, data center, etc.) →

Continent Country Data Center Owner Node Provider Node Operator
EXISTING 7 1 1 1 1 1
PROPOSED 6 (-14.29%) 1 1 1 1 1

See here for acceptable limits → Motion 132136

The above subnet information is illustrated below, followed by a node reference table:

Map Description
  • Red marker represents a removed node (transparent center for overlap visibility)
  • Green marker represents an added node
  • Blue marker represents an unchanged node
  • Highlighted patches represent the country the above nodes sit within (red if the country is removed, green if added, otherwise grey)
  • Light grey markers with yellow borders are examples of unassigned nodes that would be viable candidates for joining the subnet according to formal decentralisation coefficients (so this proposal can be viewed in the context of alternative solutions that are not being used)

Node Changes
Action Node Status Continent Country Data Center Owner Node Provider Node Operator
Remove mehhn UP :bar_chart: Europe Belgium Antwerp (an1) Datacenter United Allusion pgunx
Remove rptzo UP :bar_chart: Asia Singapore Singapore 3 (sg3) Racks Central OneSixtyTwo Digital Capital 5mhxl
Add afjfz UNASSIGNED :bar_chart: Americas Canada Toronto 2 (to2) Cyxtera Blockchain Development Labs 4lp6i
Add 2yzcy UNASSIGNED :bar_chart: Asia Singapore Singapore (sg1) Telin OneSixtyTwo Digital Capital d4bin
Other Nodes
Node Status Continent Country Data Center Owner Node Provider Node Operator
r4dbq UP :bar_chart: Oceania Australia Queensland 1 (sc1) NEXTDC ANYPOINT PTY LTD srrm2
4mm7j UP :bar_chart: Europe Germany Geneva 2 (ge2) SafeHost Archery Blockchain SCSp 5atxd
jj7si UP :bar_chart: Europe Spain Madrid 1 (ma1) Ginernet Artem Horodyskyi vgi2b
xqhoe UP :bar_chart: Asia Georgia Tbilisi 1 (tb1) Cloud9 George Bassadone yhfy4
rzm7j UP :bar_chart: Europe Croatia Zagreb 1 (zg1) Anonstake Anonstake 3sm7v
vwsvq UP :bar_chart: Asia Israel Tel Aviv 1 (tv1) Interhost GeoNodes LLC lis4o
w53hu UP :bar_chart: Asia Japan Tokyo 2 (ty2) Equinix Starbase dpt4y
vuizy UP :bar_chart: Europe Romania Bucharest (bu1) M247 Iancu Aurel c5ssg
efdju UP :bar_chart: Europe Slovenia Maribor (mb1) Posita.si Fractal Labs AG 3xiew
tgmtp UP :bar_chart: Europe Sweden Stockholm 1 (sh1) Digital Realty DFINITY Stiftung lgp6d
ijqzn UP :bar_chart: Americas United States of America (the) Allentown (aw1) Tierpoint Bigger Capital codio

*This comment references the latest comment in the Subnet Management - General Discussion thread only to generate an automated cross-link from the general thread (to improve topic navigation).


You may wish to follow D-QUORUM if you found this analysis helpful.

Known Neurons to follow if you're too busy to keep on top of things like this

If you found this analysis helpful and would like to follow the vote of the LORIMER known neuron in the future, consider configuring LORIMER as a followee for the Subnet Management topic.

Additional good neurons to follow:

  • D-QUORUM (a highly decentralized neuron that follows neurons that have been elected by the NNS)
  • Synapse (currently follows the LORIMER and CodeGov known neurons for Subnet Management, and is a generally well informed known neuron to follow on numerous other topics)
  • CodeGov (actively reviews and votes on Subnet Management proposals, and is well informed on numerous other technical topics)
  • WaterNeuron (the WaterNeuron DAO frequently discuss proposals like this in order to vote responsibly based on DAO consensus)

Note that this analysis involved data provided by the IC-API, which is not open source. I’m in the process of switching over to more verifiable sources of this sort of information for future proposal reviews. See here for related discussion.

Voted to adopt proposal #134541.

The proposal aims to replace two cordoned healthy Active status node mehhn from the AN1 Data Center in Belgium, and cordoned healthy Active status node rptzo from the SG3 Data Center in Singapore, with unassigned healthy Awaiting status node afjfz from Toronto2 and with unassigned healthy Awaiting status node 2yzcy from Singapore, without any change to the decentralization of the subnet.
The motivation makes sense and the provided Forum link included in the summary provides further info, also it can be checked here.