This topic is intended to capture Subnet Management activities over time for the w4rem subnet, providing a place to ask questions and make observations about the management of this subnet.
DFINITY will submit a proposal today to reduce the notarization delay on the subnet, w4rem, similar to what has happened on other subnets in recent weeks (you can find all details in this forum thread).
1 Like
The table below shows the trustworthy metrics failure rate for nodes on the w4rem subnet.
The has been a step change on the 28th of October. The failure rate for all nodes on this subnet increased to unacceptable high levels. Rewards for some of the node providers on this subnet will be deducted if this trend continues.
It looks like something on the subnet or on protocol level changed because all the node on this subnet are experiencing the same symptoms. Even the nodes in Belguim’s failure rate increased overnight from 0.08% to 2.24%?
It looks also if the effect is much greater on nodes far away from the “majority” - Costa Rica (48%), Australia (19%), Singapore(14%), South Africa(11%). Could this be a network latency issue?
Do you have any idea what is going on here?
@dsharifi @Lorimer @sat - your thoughts please
1 Like
Nice observations 
I noted a little while ago when the notarisation delay was reduced on a subnet that it caused one of the existing nodes to need swapping out of the subnet because it was unable to keep up with the new block rate. Essentially it’s performance hadn’t changed, but relative to the other nodes that were now producing blocks faster, the node looked much worse and was considered degraded.
In summary, the reduction in notarisation delay that has occurred now on all subnets can result in a situation where node performance has not reduced in absolute terms, but has reduced in relative terms.
Does that make sense?
@Lorimer Your reply doesn’t make sense:
The changes you referred to on this subnet have been made on Sep-12 - this is something else.
Maybe the Dfinity engineers can point the node providers in the right direction.
Regardless, we can’t have subnet or protocol settings that causes 4 out of 13 nodes not be able to keep up with the new block rate.
4 out of 13 node providers on this subnet will be penalised due to a block failure rate of more that 10% and this looks like a protocol issue.
Please see the node id’s for your reference:
The point I’m making isn’t unique to one type of performance improvement proposal. Improvements are being made all the time. My point is simply that this can expose nodes that are unable to keep up with the rest. It’s happened numerous times before.
I accept that this may not be the case here. I haven’t looked into this specific case.
From the 28th on, we started deploying https://dashboard.internetcomputer.org/release/75dd48c38f296fc907c269263f96633fa8a29d0e to subnets. However, this version landed on w4rem on 2024-10-31, 9:21:39 AM UTC. And I don’t see the error rate spiking on this subnet on the 31st compared to the 30th. The error rate spiked the day after the subnet upgrade to the new version - so on the 1st of Nov.
It could still be the case that this new version introduces an issue or makes the problems more likely, but it certainly isn’t obvious to me.
@sat @bjoernek @tina23 @dsharifi @icarus @Lorimer @wpb @SvenF @katiep
The node provider blockmaker failure rate on this subnet remains a concern.
The table below shows the average failure rate for all the nodes on the subnet. Some protocol change were made in October that affects all the node providers on this subnet.
The table below shows the relation between average latency to other nodes on the subnet and the node provider blockmaker failure rate:
Do you share my concerns?
2 Likes
Thank you @Lerak for sharing the data, we will have a look!
Regarding elevated failure rates affecting the whole subnet: As discussed in the November node provider working group, the delta between a node’s performance and the average or median failure rate of all nodes in a subnet might be a good approach to capture such cases; we will review in more detail.
7 Likes
Yes, indeed. However for Costa Rica we also have a data center problem upstream that is being worked on. I hope it will be resolved soon.
1 Like
Replace nodes in subnet w4rem
Motivation:
The following nodes in subnet w4rem have been cordoned and need to be removed from the subnet:
- af7ti: offboarding BO1 DC after 48 months; DFINITY Gen-1 servers available for sale through silent auction
- i6baw: offboarding AN1 DC after 48 months; Proposal: Update Interim Gen-1 Node Provider Remuneration After 48 months - #44 by Paul_DC and Proposal: Update Interim Gen-1 Node Provider Remuneration After 48 months - #46 by katiep
Decentralization Nakamoto coefficient changes for subnet w4rem-dv5e3-widiz-wbpea-kbttk-mnzfm-tzrc7-svcj3-kbxyb-zamch-hqe:
node_provider: 5.00 -> 5.00 (+0%)
data_center: 5.00 -> 5.00 (+0%)
data_center_owner: 5.00 -> 5.00 (+0%)
area: 5.00 -> 5.00 (+0%)
country: 5.00 -> 5.00 (+0%)
Mean Nakamoto comparison: 5.00 → 5.00 (+0%)
Overall replacement impact: equal decentralization across all features
Details
Nodes removed:
af7ti-auyik-jfsne-tljmz-6purg-2msmy-jw34z-b4ie3-abk5f-h23xt-zae[health: healthy]i6baw-ywtnm-q36ab-zi7o2-f5bva-xji6q-puhpm-pb73u-qlc3e-3nemh-kae[health: healthy]
Nodes added:
7pwmx-4zsiq-saplf-kl2sd-4yvr3-la2yi-artrs-m7voc-htak4-fii5y-sqe[health: healthy]hf6fg-j4n43-ufjh4-qzwo5-vtk6g-b6pvt-dyxs4-6toq3-fnmle-y67at-rqe[health: healthy]
node_provider data_center data_center_owner area country
------------- ----------- ----------------- ---- -------
4jjya-hlyyc-s766p-fd6gr-d6tvv-vo3ah-j5ptx-i73gw-mwgyd-rw6w2-rae 1 an1 1 -> 0 Cloud9 1 Bucuresti 1 AU 1
6nbcy-kprg6-ax3db-kh3cz-7jllk-oceyh-jznhs-riguq-fvk6z-6tsds-rqe 1 bo1 1 -> 0 DEAC 1 Cape Town 1 BE 1 -> 0
6r5lw-l7db7-uwixn-iw5en-yy55y-ilbtq-e6gcv-g22r2-j3g6q-y37jk-jqe 1 bu1 1 Datacenter United 1 -> 0 Flanders 1 -> 0 CA 1
6sq7t-knkul-fko6h-xzvnf-ktbvr-jhx7r-hapzr-kjlek-whugy-zt6ip-xqe 1 cr1 1 Digital Realty 0 -> 1 Florida 0 -> 1 CH 1
bvcsg-3od6r-jnydw-eysln-aql7w-td5zn-ay5m6-sibd2-jzojt-anwag-mqe 0 -> 1 ct2 1 Equinix 1 Massachusetts 1 -> 0 CR 1
i7dto-bgkj2-xo5dx-cyrb7-zkk5y-q46eh-gz6iq-qkgyc-w4qte-scgtb-6ae 1 jv1 0 -> 1 Green.ch 1 Melbourne 1 GE 1
ihbuj-erwnc-tkjux-tqtnv-zkoar-uniy2-sk2go-xfpkc-znbb4-seukm-wqe 1 mn2 1 INAP 1 -> 0 Panvel 1 IN 1
lq5ra-f4ibl-t7wpy-hennc-m4eb7-tnfxe-eorgd-onpsl-wervo-7chjj-6qe 1 -> 0 mtl1 1 Leaseweb 1 Quebec 1 JP 1
py2kr-ipr2p-ryh66-x3a3v-5ts6u-7rfhf-alkna-ueffh-hz5ox-lt6du-qqe 1 pl2 1 M247 1 Riga 1 LV 1
rbn2y-6vfsb-gv35j-4cyvy-pzbdu-e5aum-jzjg6-5b4n5-vuguf-ycubq-zae 1 -> 0 rg1 1 NEXTDC 1 San Jose 1 RO 1
sixix-2nyqd-t2k2v-vlsyz-dssko-ls4hl-hyij4-y7mdp-ja6cj-nsmpf-yae 1 sg1 1 Navegalo 1 Singapore 1 SE 0 -> 1
spp3m-vawt7-3gyh6-pjz5d-6zidf-up3qb-yte62-otexv-vfpqg-n6awf-lqe 0 -> 1 sh1 0 -> 1 Telin 1 Stockholm 0 -> 1 SG 1
ulyfm-vkxtj-o42dg-e4nam-l4tzf-37wci-ggntw-4ma7y-d267g-ywxi6-iae 1 tb1 1 Teraco 1 Tbilisi 1 US 1
vegae-c4chr-aetfj-7gzuh-c23sx-u2paz-vmvbn-bcage-pu7lu-mptnn-eqe 1 ty1 1 Tierpoint 0 -> 1 Tokyo 1 ZA 1
zy4m7-z5mhs-zfkpl-zlsjl-blrbx-mvvmq-5z4zu-mf7eq-hhv7o-ezfro-3ae 1 zh5 1 Yotta 1 Zurich 1
1 Like
A new proposal with id 134581 has been submitted, please take a look.
Click here to open proposal details
Replace nodes in subnet w4rem
Motivation:
The following nodes in subnet w4rem have been cordoned and need to be removed from the subnet:
- af7ti: offboarding BO1 DC after 48 months; DFINITY Gen-1 servers available for sale through silent auction
- i6baw: offboarding AN1 DC after 48 months; Proposal: Update Interim Gen-1 Node Provider Remuneration After 48 months - #44 by Paul_DC and Proposal: Update Interim Gen-1 Node Provider Remuneration After 48 months - #46 by katiep
Decentralization Nakamoto coefficient changes for subnet w4rem-dv5e3-widiz-wbpea-kbttk-mnzfm-tzrc7-svcj3-kbxyb-zamch-hqe:
node_provider: 5.00 -> 5.00 (+0%)
data_center: 5.00 -> 5.00 (+0%)
data_center_owner: 5.00 -> 5.00 (+0%)
area: 5.00 -> 5.00 (+0%)
country: 5.00 -> 5.00 (+0%)
Mean Nakamoto comparison: 5.00 → 5.00 (+0%)
Overall replacement impact: equal decentralization across all features
Details
Nodes removed:
af7ti-auyik-jfsne-tljmz-6purg-2msmy-jw34z-b4ie3-abk5f-h23xt-zae[health: healthy]i6baw-ywtnm-q36ab-zi7o2-f5bva-xji6q-puhpm-pb73u-qlc3e-3nemh-kae[health: healthy]
Nodes added:
7pwmx-4zsiq-saplf-kl2sd-4yvr3-la2yi-artrs-m7voc-htak4-fii5y-sqe[health: healthy]fe7vt-yjpwr-js7dn-62rht-w46qq-kbxls-ujd3f-bgotr-hjdzy-wu5kq-vae[health: healthy]
node_provider data_center data_center_owner area country
------------- ----------- ----------------- ---- -------
4jjya-hlyyc-s766p-fd6gr-d6tvv-vo3ah-j5ptx-i73gw-mwgyd-rw6w2-rae 1 an1 1 -> 0 Cloud9 1 Arizona 0 -> 1 AU 1
6nbcy-kprg6-ax3db-kh3cz-7jllk-oceyh-jznhs-riguq-fvk6z-6tsds-rqe 1 bo1 1 -> 0 CyrusOne 0 -> 1 Bucuresti 1 BE 1 -> 0
6r5lw-l7db7-uwixn-iw5en-yy55y-ilbtq-e6gcv-g22r2-j3g6q-y37jk-jqe 1 bu1 1 DEAC 1 Cape Town 1 CA 1
6sq7t-knkul-fko6h-xzvnf-ktbvr-jhx7r-hapzr-kjlek-whugy-zt6ip-xqe 1 cr1 1 Datacenter United 1 -> 0 Flanders 1 -> 0 CH 1
bvcsg-3od6r-jnydw-eysln-aql7w-td5zn-ay5m6-sibd2-jzojt-anwag-mqe 0 -> 1 ct2 1 Digital Realty 0 -> 1 Massachusetts 1 -> 0 CR 1
i7dto-bgkj2-xo5dx-cyrb7-zkk5y-q46eh-gz6iq-qkgyc-w4qte-scgtb-6ae 1 mn2 1 Equinix 1 Melbourne 1 GE 1
ihbuj-erwnc-tkjux-tqtnv-zkoar-uniy2-sk2go-xfpkc-znbb4-seukm-wqe 1 mtl1 1 Green.ch 1 Panvel 1 IN 1
izmhk-lpjum-uo4oy-lviba-yctpc-arg4b-2ywim-vgoiu-gqaj2-gskmw-2qe 0 -> 1 ph1 0 -> 1 INAP 1 -> 0 Quebec 1 JP 1
lq5ra-f4ibl-t7wpy-hennc-m4eb7-tnfxe-eorgd-onpsl-wervo-7chjj-6qe 1 -> 0 pl2 1 Leaseweb 1 Riga 1 LV 1
py2kr-ipr2p-ryh66-x3a3v-5ts6u-7rfhf-alkna-ueffh-hz5ox-lt6du-qqe 1 rg1 1 M247 1 San Jose 1 RO 1
rbn2y-6vfsb-gv35j-4cyvy-pzbdu-e5aum-jzjg6-5b4n5-vuguf-ycubq-zae 1 -> 0 sg1 1 NEXTDC 1 Singapore 1 SE 0 -> 1
sixix-2nyqd-t2k2v-vlsyz-dssko-ls4hl-hyij4-y7mdp-ja6cj-nsmpf-yae 1 sh1 0 -> 1 Navegalo 1 Stockholm 0 -> 1 SG 1
ulyfm-vkxtj-o42dg-e4nam-l4tzf-37wci-ggntw-4ma7y-d267g-ywxi6-iae 1 tb1 1 Telin 1 Tbilisi 1 US 1
vegae-c4chr-aetfj-7gzuh-c23sx-u2paz-vmvbn-bcage-pu7lu-mptnn-eqe 1 ty1 1 Teraco 1 Tokyo 1 ZA 1
zy4m7-z5mhs-zfkpl-zlsjl-blrbx-mvvmq-5z4zu-mf7eq-hhv7o-ezfro-3ae 1 zh5 1 Yotta 1 Zurich 1
Voted to reject proposal 134581 as this is part of a large batch of non-critical proposals timed such that the voting period clashes with national holidays, thereby allowing insufficient time for an appropriately detailed review to take place.
1 Like
Rejected proposal 134581. There’s no time to review this and the 18 other proposals in the same batch properly over Xmas eve, Xmas and Boxing Day. This is a non-critical proposal.
More information and discussion on this thread.
Proposal 134581
Vote: ADOPT
Replaces cordoned nodes af7ti and i6baw with nodes 7pwmx and fe7vt on subnet w4rem.
The reason for this proposal is to offboard BO1 and AN1 DCs consistent with forum posts made on the forum thread used for posts regarding the renovation/sell of Gen-1 node machines by NPs.
Both the NP and DC stated in the forum post and forum posts match the ones from the node being removed in the proposal.
2 Likes
Voted to adopt proposal #134581.
The proposal replaces two cordoned nodes, one healthy Active status node af7ti from the BO1 Data Center in Massachusetts, and cordoned healthy Active status node i6baw from the AN1 Data Center in Belgium, with unassigned healthy Awaiting status node 7pwmx from Sweden and with unassigned healthy Awaiting status node fe7vt from Arizona, without any change to the decentralization of the subnet.
The motivation makes sense and the provided Forum link included in the summary provides further info, also it can be checked here.
1 Like
During our latest NP working group meeting, a point was raised regarding a potential positive correlation between latency and failure rates across subnets.
To explore this relationship, we analyzed data from multiple subnets over the period from April 2024 to December 2024. The approach involved the following steps:
- Distance Calculation: For each node within a subnet, we computed the average geographical distance to its five closest nodes.
- Latency Buckets: These average distances were bucketized into three categories to approximate latency levels:
- Low latency: 0 to 1000 km
- Mid latency: 1000 to 3000 km
- High latency: ≥ 3000 km
- Comparison: The median failure rate for nodes in each bucket was calculated and plotted for comparison.
Our analysis highlighted varying results across subnets. The subnet that triggered this investigation, identified as w4rem, exhibited a positive correlation between latency and failure rate starting in October.
However, when we expanded the analysis to include other subnets, the observed correlation was inconsistent and in some cases even negative.
Conclusion
The broader analysis across subnets does not support a consistent correlation.
- The correlation between latency and failure rate may be workload dependent.
- The issue does not warrant additional measures or special treatments beyond those already in place (10% buffer).
- The existing 10% discount buffer applied to deviations from the 75th percentile of failure rates remains sufficient to mitigate potential issues related to latency.
5 Likes
I always appreciate how DFINITY consistently makes data based decisions. This is a very interesting analysis. Thanks for sharing.
1 Like
Replace a node in subnet w4rem
Motivation:
The following nodes in subnet w4rem have been cordoned and need to be removed from the subnet:
- jiy6l: offboarding ZH5 DC after 48 months; Proposal: Update Interim Gen-1 Node Provider Remuneration After 48 months - #117 by Sygnum and Proposal: Update Interim Gen-1 Node Provider Remuneration After 48 months - #118 by katiep
Decentralization Nakamoto coefficient changes for subnet w4rem-dv5e3-widiz-wbpea-kbttk-mnzfm-tzrc7-svcj3-kbxyb-zamch-hqe:
node_provider: 5.00 -> 5.00 (+0%)
data_center: 5.00 -> 5.00 (+0%)
data_center_owner: 5.00 -> 5.00 (+0%)
area: 5.00 -> 5.00 (+0%)
country: 5.00 -> 5.00 (+0%)
Mean Nakamoto comparison: 5.00 → 5.00 (+0%)
Overall replacement impact: equal decentralization across all features
Details
Nodes removed:
jiy6l-jx7ea-k7y2u-tznfb-dcrm2-5fhrt-r63pd-wdfse-2ubgq-6pfvi-lae[health: healthy]
Nodes added:
rpmaq-iinj7-yufzr-4ekre-4sbok-wzkzu-guxeq-sxmvi-5rqm4-3kfe2-oae[health: healthy]
node_provider data_center data_center_owner area country
------------- ----------- ----------------- ---- -------
4jjya-hlyyc-s766p-fd6gr-d6tvv-vo3ah-j5ptx-i73gw-mwgyd-rw6w2-rae 1 bu1 1 Cloud9 1 Arizona 1 AU 1
6nbcy-kprg6-ax3db-kh3cz-7jllk-oceyh-jznhs-riguq-fvk6z-6tsds-rqe 1 cr1 1 CyrusOne 1 Bucuresti 1 CA 1
6r5lw-l7db7-uwixn-iw5en-yy55y-ilbtq-e6gcv-g22r2-j3g6q-y37jk-jqe 1 ct2 1 DEAC 1 Cape Town 1 CH 1
6sq7t-knkul-fko6h-xzvnf-ktbvr-jhx7r-hapzr-kjlek-whugy-zt6ip-xqe 1 mn2 1 Digital Realty 1 Melbourne 1 CR 1
bvcsg-3od6r-jnydw-eysln-aql7w-td5zn-ay5m6-sibd2-jzojt-anwag-mqe 1 mtl1 1 Equinix 1 Panvel 1 GE 1
i7dto-bgkj2-xo5dx-cyrb7-zkk5y-q46eh-gz6iq-qkgyc-w4qte-scgtb-6ae 1 ph1 1 Green.ch 1 Quebec 1 IN 1
ihbuj-erwnc-tkjux-tqtnv-zkoar-uniy2-sk2go-xfpkc-znbb4-seukm-wqe 1 pl2 1 Leaseweb 1 Riga 1 JP 1
izmhk-lpjum-uo4oy-lviba-yctpc-arg4b-2ywim-vgoiu-gqaj2-gskmw-2qe 1 rg1 1 M247 1 San Jose 1 LV 1
py2kr-ipr2p-ryh66-x3a3v-5ts6u-7rfhf-alkna-ueffh-hz5ox-lt6du-qqe 1 sg1 1 NEXTDC 1 Singapore 1 RO 1
sixix-2nyqd-t2k2v-vlsyz-dssko-ls4hl-hyij4-y7mdp-ja6cj-nsmpf-yae 1 sh1 1 Navegalo 1 Stockholm 1 SE 1
ulyfm-vkxtj-o42dg-e4nam-l4tzf-37wci-ggntw-4ma7y-d267g-ywxi6-iae 1 tb1 1 Telin 1 Tbilisi 1 SG 1
vegae-c4chr-aetfj-7gzuh-c23sx-u2paz-vmvbn-bcage-pu7lu-mptnn-eqe 1 ty1 1 Teraco 1 Tokyo 1 US 1
zy4m7-z5mhs-zfkpl-zlsjl-blrbx-mvvmq-5z4zu-mf7eq-hhv7o-ezfro-3ae 1 zh5 1 -> 0 Yotta 1 Zurich 1 ZA 1
zh6 0 -> 1
Replace a node in subnet w4rem
Motivation:
The following nodes in subnet w4rem have been cordoned and need to be removed from the subnet:
- jiy6l: offboarding ZH5 DC after 48 months; Proposal: Update Interim Gen-1 Node Provider Remuneration After 48 months - #117 by Sygnum and Proposal: Update Interim Gen-1 Node Provider Remuneration After 48 months - #118 by katiep
Decentralization Nakamoto coefficient changes for subnet w4rem-dv5e3-widiz-wbpea-kbttk-mnzfm-tzrc7-svcj3-kbxyb-zamch-hqe:
node_provider: 5.00 -> 5.00 (+0%)
data_center: 5.00 -> 5.00 (+0%)
data_center_owner: 5.00 -> 5.00 (+0%)
area: 5.00 -> 5.00 (+0%)
country: 5.00 -> 5.00 (+0%)
Mean Nakamoto comparison: 5.00 → 5.00 (+0%)
Overall replacement impact: equal decentralization across all features
Details
Nodes removed:
jiy6l-jx7ea-k7y2u-tznfb-dcrm2-5fhrt-r63pd-wdfse-2ubgq-6pfvi-lae[health: healthy]
Nodes added:
udlch-g42nv-er23m-bgpxk-oiymd-knkro-2dczg-hyv4x-2yiw6-q2jcz-hqe[health: healthy]
node_provider data_center data_center_owner area country
------------- ----------- ----------------- ---- -------
4jjya-hlyyc-s766p-fd6gr-d6tvv-vo3ah-j5ptx-i73gw-mwgyd-rw6w2-rae 1 bu1 1 Cloud9 1 Arizona 1 AU 1
6nbcy-kprg6-ax3db-kh3cz-7jllk-oceyh-jznhs-riguq-fvk6z-6tsds-rqe 1 cr1 1 CyrusOne 1 Bucuresti 1 CA 1
6r5lw-l7db7-uwixn-iw5en-yy55y-ilbtq-e6gcv-g22r2-j3g6q-y37jk-jqe 1 ct2 1 DEAC 1 Cape Town 1 CH 1
6sq7t-knkul-fko6h-xzvnf-ktbvr-jhx7r-hapzr-kjlek-whugy-zt6ip-xqe 1 mn2 1 Digital Realty 1 Melbourne 1 CR 1
bvcsg-3od6r-jnydw-eysln-aql7w-td5zn-ay5m6-sibd2-jzojt-anwag-mqe 1 mtl1 1 Equinix 1 Panvel 1 GE 1
i7dto-bgkj2-xo5dx-cyrb7-zkk5y-q46eh-gz6iq-qkgyc-w4qte-scgtb-6ae 1 ph1 1 Green.ch 1 Quebec 1 IN 1
ihbuj-erwnc-tkjux-tqtnv-zkoar-uniy2-sk2go-xfpkc-znbb4-seukm-wqe 1 pl2 1 Leaseweb 1 Riga 1 JP 1
izmhk-lpjum-uo4oy-lviba-yctpc-arg4b-2ywim-vgoiu-gqaj2-gskmw-2qe 1 rg1 1 M247 1 San Jose 1 LV 1
py2kr-ipr2p-ryh66-x3a3v-5ts6u-7rfhf-alkna-ueffh-hz5ox-lt6du-qqe 1 sg1 1 NEXTDC 1 Singapore 1 RO 1
sixix-2nyqd-t2k2v-vlsyz-dssko-ls4hl-hyij4-y7mdp-ja6cj-nsmpf-yae 1 sh1 1 Navegalo 1 Stockholm 1 SE 1
ulyfm-vkxtj-o42dg-e4nam-l4tzf-37wci-ggntw-4ma7y-d267g-ywxi6-iae 1 tb1 1 Telin 1 Tbilisi 1 SG 1
vegae-c4chr-aetfj-7gzuh-c23sx-u2paz-vmvbn-bcage-pu7lu-mptnn-eqe 1 ty1 1 Teraco 1 Tokyo 1 US 1
zy4m7-z5mhs-zfkpl-zlsjl-blrbx-mvvmq-5z4zu-mf7eq-hhv7o-ezfro-3ae 1 zh5 1 -> 0 Yotta 1 Zurich 1 ZA 1
zh6 0 -> 1