Thanks @Daniel_Web3 we’re monitoring that and notifying the node providers as well, and have weekly node replacements submitted every Friday unless there are critical issues with many nodes at the same time. In this situation, the outages seems to be coincidental and not related to the issue last week.
1 Like
Came across this just now which is useful/interesting context
Voted to adopt proposal 134278. The proposal seeks to remove a cordoned node from the subnet and specifies that the associated data centre is being offboarded “after 48 months”. Decentralisation parameters are unchanged. The necessary context is provided by this forum post and associated discussion. For future proposals of this type I recommend that the background context be included in the proposal text for ease of verification.
I’ve voted to reject proposal 134278. It makes claims that I see no clear way of verifying, and supporting statements regarding the ownership of the data centre are inconsistent with records held in the IC registry. The proposal was announced here.
Voted to adopt proposal 134278. This proposal is part of a sequence of steps to remove cordoned nodes from subnets as the associated data centeres are being offboarded after 48 months of their respective DC contracts that are still private and were signed up before the Genesis. There is a great and detailed explanation of this changes in this forum post and the forum thread it is in. In the wiki there is a series of Steps for Gen-1 Node onboarding after 48 months that need to be followed in order for the nodes to continue earning rewards which starts by making a forum post in the following thread. As we can verify no one as come forward with nodes from the DCs in this proposals so I don’t see any issues with the removal of this nodes.
A new proposal with id 134550 has been submitted, please take a look.
Click here to open proposal details
Replace nodes in subnet lhg73
Motivation:
The following nodes in subnet lhg73 have been cordoned and need to be removed from the subnet:
- mihvd: offboarding AN1 DC after 48 months; Proposal: Update Interim Gen-1 Node Provider Remuneration After 48 months - #44 by Paul_DC and Proposal: Update Interim Gen-1 Node Provider Remuneration After 48 months - #46 by katiep
- vlbpb: offboarding MR1 DC after 48 months; DFINITY Gen-1 servers available for sale through silent auction
Decentralization Nakamoto coefficient changes for subnet lhg73-sax6z-2zank-6oer2-575lz-zgbxx-ptudx-5korm-fy7we-kh4hl-pqe:
node_provider: 5.00 -> 5.00 (+0%)
data_center: 5.00 -> 5.00 (+0%)
data_center_owner: 5.00 -> 5.00 (+0%)
area: 5.00 -> 5.00 (+0%)
country: 5.00 -> 5.00 (+0%)
Mean Nakamoto comparison: 5.00 → 5.00 (+0%)
Overall replacement impact: equal decentralization across all features
Details
Nodes removed:
mihvd-umv3j-cjsl2-bfsdu-td7aw-2y6if-aw4fn-cghkm-v2oxd-kj75q-cae[health: healthy]vlbpb-szwgu-jqul3-kpfje-ozgkp-jdwz4-f2hhq-6w6tf-e2lrl-4bmad-xae[health: healthy]
Nodes added:
u6j47-wepye-qtjf3-vkvnv-q6nfc-a3wky-grxp6-wbjuy-xejgs-qalg4-4ae[health: healthy]23kbh-5xqrz-2avco-tddeh-p6724-shtvz-rg3nq-d7rhu-ztedb-vki46-dqe[health: healthy]
node_provider data_center data_center_owner area country
------------- ----------- ----------------- ---- -------
3oqw6-vmpk2-mlwlx-52z5x-e3p7u-fjlcw-yxc34-lf2zq-6ub2f-v63hk-lae 1 an1 1 -> 0 Africa Data Centres 1 Bogota 1 AU 1
6nbcy-kprg6-ax3db-kh3cz-7jllk-oceyh-jznhs-riguq-fvk6z-6tsds-rqe 1 aw1 1 Data Inn 1 Flanders 1 -> 0 BE 1 -> 0
7a4u2-gevsy-5c5fs-hsgri-n2kdz-dxxwf-btcfp-jykro-l4y7c-7xky2-aqe 1 bg1 1 Datacenter United 1 -> 0 Gauteng 1 CH 1
7ryes-jnj73-bsyu4-lo6h7-lbxk5-x4ien-lylws-5qwzl-hxd5f-xjh3w-mqe 1 ge1 1 Digital Realty 1 Geneva 1 CO 1
bvcsg-3od6r-jnydw-eysln-aql7w-td5zn-ay5m6-sibd2-jzojt-anwag-mqe 1 hk1 1 EdgeUno 1 HongKong 1 EE 1
fwnmn-zn7yt-5jaia-fkxlr-dzwyu-keguq-npfxq-mc72w-exeae-n5thj-oae 1 jb2 1 Equinix 0 -> 1 Ljubljana 1 FR 1 -> 0
nmdd6-rouxw-55leh-wcbkn-kejit-njvje-p4s6e-v64d3-nlbjb-vipul-mae 1 lj1 1 HighDC 1 Navi Mumbai 1 HK 1
otzuu-dldzs-avvu2-qwowd-hdj73-aocy7-lacgi-carzj-m6f2r-ffluy-fae 1 mr1 1 -> 0 NEXTDC 1 Pennsylvania 1 IN 1
r3yjn-kthmg-pfgmb-2fngg-5c7d7-t6kqg-wi37r-j7gy6-iee64-kjdja-jae 1 nm1 1 Posita.si 1 Provence-Alpes-Cote d'Azur 1 -> 0 JP 0 -> 1
rbn2y-6vfsb-gv35j-4cyvy-pzbdu-e5aum-jzjg6-5b4n5-vuguf-ycubq-zae 1 -> 0 sc1 1 Rivram 1 Queensland 1 LT 1
sixix-2nyqd-t2k2v-vlsyz-dssko-ls4hl-hyij4-y7mdp-ja6cj-nsmpf-yae 0 -> 1 sg2 1 Telia DC 1 Singapore 1 SE 0 -> 1
ulyfm-vkxtj-o42dg-e4nam-l4tzf-37wci-ggntw-4ma7y-d267g-ywxi6-iae 1 sh1 0 -> 1 Telin 1 Stockholm 0 -> 1 SG 1
vegae-c4chr-aetfj-7gzuh-c23sx-u2paz-vmvbn-bcage-pu7lu-mptnn-eqe 1 ta2 1 Tierpoint 1 Tallinn 1 SI 1
wdjjk-blh44-lxm74-ojj43-rvgf4-j5rie-nm6xs-xvnuv-j3ptn-25t4v-6ae 1 ty1 0 -> 1 Unicom 1 Tokyo 0 -> 1 US 1
vl2 1 Vilnius 1 ZA 1
1 Like
Proposal 134550
TLDR: I’ll adopt. The proposal links directly to what appears to be discussion with the NP about the proposal. Decentralisation stats look good.
Decentralisation Stats
Subnet node distance stats (distance between any 2 nodes in the subnet) →
| Smallest Distance | Average Distance | Largest Distance | |
|---|---|---|---|
| EXISTING | 317.676 km | 8096.384 km | 18505.029 km |
| PROPOSED | 549.609 km (+73%) | 8476.115 km (+4.7%) | 18505.029 km |
This proposal slightly increases decentralisation, considered purely in terms of geographic distance (and therefore there’s a slight theoretical increase in localised disaster resilience). 
Subnet characteristic counts →
| Continents | Countries | Data Centers | Owners | Node Providers | Node Operator | |
|---|---|---|---|---|---|---|
| EXISTING | 5 | 13 | 13 | 13 | 13 | 13 |
| PROPOSED | 5 | 13 | 13 | 13 | 13 | 13 |
Largest number of nodes with the same characteristic (e.g. continent, country, data center, etc.) →
| Continent | Country | Data Center | Owner | Node Provider | Node Operator | |
|---|---|---|---|---|---|---|
| EXISTING | 6 | 1 | 1 | 1 | 1 | 1 |
| PROPOSED | 5 (-16.67%) | 1 | 1 | 1 | 1 | 1 |
See here for acceptable limits → Motion 132136
The above subnet information is illustrated below, followed by a node reference table:
Map Description
- Red marker represents a removed node (transparent center for overlap visibility)
- Green marker represents an added node
- Blue marker represents an unchanged node
- Highlighted patches represent the country the above nodes sit within (red if the country is removed, green if added, otherwise grey)
- Light grey markers with yellow borders are examples of unassigned nodes that would be viable candidates for joining the subnet according to formal decentralisation coefficients (so this proposal can be viewed in the context of alternative solutions that are not being used)
Node Changes
| Action | Node | Status | Continent | Country | Data Center | Owner | Node Provider | Node Operator | |
|---|---|---|---|---|---|---|---|---|---|
| Remove |  |
||||||||
| Remove | |
||||||||
| Add | 23kbh | UNASSIGNED | |
Asia | Japan | Tokyo (ty1) | Equinix | Starbase | cqjev |
| Add | u6j47 | UNASSIGNED | |
Europe | Sweden | Stockholm 1 (sh1) | Digital Realty | DFINITY Stiftung | lgp6d |
Other Nodes
| Node | Status | Continent | Country | Data Center | Owner | Node Provider | Node Operator | |
|---|---|---|---|---|---|---|---|---|
| 56ovz | UP | |
Oceania | Australia | Queensland 1 (sc1) | NEXTDC | ANYPOINT PTY LTD | srrm2 |
| c3xxv | UP | |
Europe | Switzerland | Geneva (ge1) | HighDC | Archery Blockchain SCSp | yngfj |
| r4642 | UP | |
Asia | China | HongKong 1 (hk1) | Unicom | Pindar Technology Limited | vzsx4 |
| ihttm | UP | |
Americas | Costa Rica | Bogota 1 (bg1) | EdgeUno | Geeta Kalwani | 74vhn |
| bptaj | UP | |
Europe | Estonia | Tallinn 2 (ta2) | Telia DC | Vladyslav Popov | y3nsb |
| pdo46 | UP | |
Asia | India | Navi Mumbai 1 (nm1) | Rivram | Rivram Inc | mpmyf |
| um4so | UP | |
Europe | Lithuania | Vilnius 2 (vl2) | Data Inn | George Bassadone | inluf |
| cpywp | UP | |
Asia | Singapore | Singapore 2 (sg2) | Telin | OneSixtyTwo Digital Capital | qffmn |
| uzy2p | UP | |
Europe | Slovenia | Ljubljana (lj1) | Posita.si | Fractal Labs AG | gl27f |
| g4eis | UP | |
Americas | United States of America (the) | Allentown (aw1) | Tierpoint | Bigger Capital | codio |
| 5v4on | UP | |
Africa | South Africa | Gauteng 2 (jb2) | Africa Data Centres | Honeycomb Capital (Pty) Ltd | 3bohy |
*This comment references the latest comment in the Subnet Management - General Discussion thread only to generate an automated cross-link from the general thread (to improve topic navigation).
You may wish to follow D-QUORUM if you found this analysis helpful.
Known Neurons to follow if you're too busy to keep on top of things like this
If you found this analysis helpful and would like to follow the vote of the LORIMER known neuron in the future, consider configuring LORIMER as a followee for the Subnet Management topic.
Additional good neurons to follow:
- D-QUORUM (a highly decentralized neuron that follows neurons that have been elected by the NNS)
- Synapse (currently follows the LORIMER and CodeGov known neurons for Subnet Management, and is a generally well informed known neuron to follow on numerous other topics)
- CodeGov (actively reviews and votes on Subnet Management proposals, and is well informed on numerous other technical topics)
- WaterNeuron (the WaterNeuron DAO frequently discuss proposals like this in order to vote responsibly based on DAO consensus)
Note that this analysis involved data provided by the IC-API, which is not open source. I’m in the process of switching over to more verifiable sources of this sort of information for future proposal reviews. See here for related discussion.
Voted to adopt proposal #134550.
The proposal replaces two cordoned healthy Active status node mihvd from the AN1 Data Center in Belgium, and cordoned healthy Active status node vlbpb from the MR1 Data Center in France, with unassigned healthy Awaiting status node u6j47 from Sweden and with unassigned healthy Awaiting status node 23kbh from Tokyo, without any change to the decentralization of the subnet.
The motivation makes sense and the provided Forum link included in the summary provides further info, also it can be checked here.
Voted to adopt proposal 134550.
This proposal replaces 2 nodes, due to offboarding AN1 and MR1 data centres. Decentralisation parameters are unchanged and remain within the requirements of the target topology. Data centre details are consistent with the links provided in the proposal.
Proposal 134550
Vote: ADOPT
Replaces cordoned nodes mihvd and vlbpb with nodes 2grd6 and 23kbh on subnet lhg73.
The reason for this proposal is to offboard MR1 and AN1 DCs consistent with forum posts made on the forum thread used for posts regarding the renovation/sell of Gen-1 node machines by NPs.
Both the NP and DC stated in the forum post match the ones from the node being removed in the proposal.
This subnet has stalled.
Proposal 134605, 134606 and 134607
Vote: ADOPT
The subnet lhg73 stalled. DFINITY made 3 proposals in order to make a simple subnet recovery but the root cause is yet to be found as explained in the following thread. DFINITY has already cast their vote in order to recover the subnet as soon as possible and is currently running with no issues, so I have voted to adopt all three proposals to recover this subnet.
3 Likes
Proposal 134605
TLDR: I’ve voted to adopt.
Halt subnet lhg73-sax6z-2zank-6oer2-575lz-zgbxx-ptudx-5korm-fy7we-kh4hl-pqe, for recovery and update ssh readonly access
This proposal has already executed, at 2024-12-26, 19:00:56 UTC (minutes after it was submitted several hours ago). As can be seen in the screenshot below, the subnet had stalled and was no longer processing blocks for a couple of hours prior to this proposal to halt the subnet.
It’s worth noting that this subnet stalled not so long ago under similar circumstances.
1 Like
Proposal 134606
TLDR : I’ve voted to reject
Update recovery CatchUp Package of subnet: lhg73 to height: 130666000
I voted to reject this CUP proposal, the reasoning for which I’ve previously discussed with DFINITY
1 Like
Proposal 134607
TLDR: I’ve voted to adopt.
Unhalt subnet lhg73-sax6z-2zank-6oer2-575lz-zgbxx-ptudx-5korm-fy7we-kh4hl-pqe, for recovery and update ssh readonly access
This proposal has already executed, at 2024-12-26, 20:43:06 UTC (minutes after it was submitted several hours ago). As can be seen in the screenshot below, after stalling, halting, CUP, and unhalting, the subnet is back on track and processing blocks (with the condition improving since the unhalt proposal executed).
1 Like
Proposal 134623
TLDR: I’ve voted to adopt.
Halt subnet lhg73-sax6z-2zank-6oer2-575lz-zgbxx-ptudx-5korm-fy7we-kh4hl-pqe, for recovery and update ssh readonly access
This proposal has already executed, at 2024-12-27, 18:09:58 UTC (minutes after it was submitted hours ago). As can be seen in the screenshot below, the subnet had stalled and was no longer processing blocks for a handful of minutes prior to this proposal to halt the subnet.
This is the third recent stall, with this last one arriving since the comment above.
Proposal 134629
TLDR : I’ve voted to reject
Update recovery CatchUp Package of subnet: lhg73 to height: 130812000
I voted to reject this CUP proposal, the reasoning for which I’ve previously discussed with DFINITY