Subnet Management - uzr34 (II)

I tried a somewhat cruder approach of using the decentralization tool to tabulate all the network nodes and then re-sorting the table by the various parameters in order to search for eligible replacement nodes. I came up with a number of possibilities, but how about these ones for example?

node_id dc_name node_provider_name node_type owner region status
7fiyj-xjxkj-fjvbl-tl4rp-6xvg3-moow7-dwtyi-nsxeg-unw7n-ivshf-pqe Cape Town 2 Kontrapunt (Pty) Ltd REPLICA Teraco Africa,ZA,Cape Town UNASSIGNED
hlc73-cklhu-zo4mt-waxb6-ew5tx-gm4or-coy2j-xnwmy-xsycn-j6f4e-iae HongKong 3 Power Meta Corporation REPLICA hkcolo Asia,HK,HongKong UNASSIGNED

I also tried using the what-if tool but got this error:

Youā€™re using an old version of the dre tool @timk11 , please run dre upgrade

Done. Thanks!

Here we go:

./dre subnet whatif uzr34-akd3s-xrdag-3ql62-ocgoh-ld2ao-tamcv-54e7j-krwgb-2gm4z-oqe --add-nodes 7fiyj-xjxkj-fjvbl-tl4rp-6xvg3-moow7-dwtyi-nsxeg-unw7n-ivshf-pqe hlc73-cklhu-zo4mt-waxb6-ew5tx-gm4or-coy2j-xnwmy-xsycn-j6f4e-iae --remove-nodes xetvj-ysqpy-wnnd4-fbytw-n6arq-w7f5e-fhsuo-7xd67-lwpt2-novnx-iae ylm4h-mc2vy-vmwvf-g2qhn-36rbw-5cunq-wpj2j-ttofo-xc27c-vfdh7-aae

Decentralization Nakamoto coefficient changes for subnet uzr34-akd3s-xrdag-3ql62-ocgoh-ld2ao-tamcv-54e7j-krwgb-2gm4z-oqe:

  node_provider: 10.00 -> 10.00    (+0%)
    data_center: 10.00 -> 10.00    (+0%)
data_center_owner: 9.00 -> 9.00    (+0%)
            city: 10.00 -> 9.00   (-10%)
          country: 6.00 -> 4.00   (-33%)
        continent: 2.00 -> 2.00    (+0%)

Mean Nakamoto comparison: 7.83 ā†’ 7.33 (-6%)

Overall replacement impact: the average log2 of Nakamoto Coefficients across all features changes from 2.7868 to 2.6640

Details

Nodes removed:
ā†’ xetvj-ysqpy-wnnd4-fbytw-n6arq-w7f5e-fhsuo-7xd67-lwpt2-novnx-iae [health: healthy, impact on decentralization: the number of different NP actors changes from 28 to 27]
ā†’ ylm4h-mc2vy-vmwvf-g2qhn-36rbw-5cunq-wpj2j-ttofo-xc27c-vfdh7-aae [health: dead, impact on decentralization: the minimum Nakamoto coefficient across all features changes from 2 to 1]

Nodes added:
++> 7fiyj-xjxkj-fjvbl-tl4rp-6xvg3-moow7-dwtyi-nsxeg-unw7n-ivshf-pqe [health: healthy, impact on decentralization: the minimum Nakamoto coefficient across all features changes from 1 to 2]
++> hlc73-cklhu-zo4mt-waxb6-ew5tx-gm4or-coy2j-xnwmy-xsycn-j6f4e-iae [health: healthy, impact on decentralization: the average log2 of Nakamoto Coefficients across all features changes from 2.7429 to 2.6640]

    node_provider                                                              data_center            data_center_owner                                    city                             country            continent

    -------------                                                              -----------            -----------------                                    ----                             -------            ---------

    4fedi-eu6ue-nd7ts-vnof5-hzg66-hgzl7-liy5n-3otyp-h7ipw-owycg-uae  0 -> 1    an1               1    Africa Data Centres                             1    Bogota                      1    AR            1    Africa         1 -> 2
    4r6qy-tljxg-slziw-zoteo-pboxh-vlctz-hkv2d-7zior-u3pxm-mmuxb-cae       1    ar1               1    Central Tower DC                                1    CABA                        1    AU            1    Asia           8 -> 7
    64xe5-tx2s3-4gjmj-pnozr-fejw2-77y5y-rhcjk-glnmx-62brf-qin5q-pqe       1    at1               1    Cloud9                                          1    California                  1    BE            1    Europe              9
    6nbcy-kprg6-ax3db-kh3cz-7jllk-oceyh-jznhs-riguq-fvk6z-6tsds-rqe       1    bg1               1    CyrusOne                                        1    Cape Town              0 -> 1    CA            1    North America       7
    6sq7t-knkul-fko6h-xzvnf-ktbvr-jhx7r-hapzr-kjlek-whugy-zt6ip-xqe  1 -> 0    bn1               1    Cyxtera                                         1    Colombo                1 -> 0    CH            1    Oceania             1
    7at4h-nhtvt-a4s55-jigss-wr2ha-ysxkn-e6w7x-7ggnm-qd3d5-ry66r-cae       1    ch3               1    Datacenter United                               1    Flanders                    1    CO            1    South America       2
    7uioy-xitfw-yqcko-5gpya-3lpsw-dw7zt-dyyyf-wfqif-jvi76-fdbkg-cqe       1    cm1          1 -> 0    Datasite                                        1    Florida                     1    CZ            1
    bvcsg-3od6r-jnydw-eysln-aql7w-td5zn-ay5m6-sibd2-jzojt-anwag-mqe       1    ct2          0 -> 1    Digital Realty                                  1    Gauteng                     1    DE            1
    dhywe-eouw6-hstpj-ahsnw-xnjxq-cmqks-47mrg-nnncb-3sr5d-rac6m-nae       1    fm1               1    EdgeUno                                         1    Georgia                     1    EE            1
    diyay-s4rfq-xnx23-zczwi-nptra-5254n-e4zn6-p7tqe-vqhzr-sd4gd-bqe       1    fr2               1    Equinix                                         2    Hesse                       1    ES            1
    eatbv-nlydd-n655c-g7j7p-gnmpz-pszdg-6e6et-veobv-ftz2y-4m752-vqe       1    hk1               1    Flexential                                      1    HongKong               1 -> 2    GE            1
    eybf4-6t6bb-unfb2-h2hhn-rrfi2-cd2vs-phksn-jdmbn-i463m-4lzds-vqe  1 -> 0    hk3          0 -> 1    Gasan                                           1    Illinois                    1    HK       1 -> 2
    fwnmn-zn7yt-5jaia-fkxlr-dzwyu-keguq-npfxq-mc72w-exeae-n5thj-oae       1    hu1               1    Ginernet                                        1    Ljubljana                   1    IL       1 -> 0
    ivf2y-crxj4-y6ewo-un35q-a7pum-wqmbw-pkepy-d6uew-bfmff-g5yxe-eae       1    jb2               1    Hurricane Electric                              1    Madrid                      1    IN            1
    izmhk-lpjum-uo4oy-lviba-yctpc-arg4b-2ywim-vgoiu-gqaj2-gskmw-2qe       1    kr2               1    Interhost                                  1 -> 0    New Delhi                   1    JP            1
    otzuu-dldzs-avvu2-qwowd-hdj73-aocy7-lacgi-carzj-m6f2r-ffluy-fae       1    lj1               1    Marvelous Web3 DC                               1    Ontario                     1    KR            1
    py2kr-ipr2p-ryh66-x3a3v-5ts6u-7rfhf-alkna-ueffh-hz5ox-lt6du-qqe  0 -> 1    ma1               1    Master Internet                                 1    Panama City                 1    LK       1 -> 0
    qdj4d-76lh3-w2q5i-kwjcd-643pq-pk42d-cziag-4hkau-35gib-m7s33-6qe       1    nd1               1    NEXTDC                                          1    Queensland                  1    PA            1
    r3yjn-kthmg-pfgmb-2fngg-5c7d7-t6kqg-wi37r-j7gy6-iee64-kjdja-jae       1    or1               1    Navegalo                                        1    Seoul                       1    PL            1
    rbn2y-6vfsb-gv35j-4cyvy-pzbdu-e5aum-jzjg6-5b4n5-vuguf-ycubq-zae       1    pc1               1    Nine.Ch                                         1    Singapore                   1    SE            1
    s5nvr-ipdxf-xg6wd-ofacm-7tl4i-nwjzx-uulum-cugwb-kbpsa-wrsgs-cae       1    sc1               1    OrionStellar                               1 -> 0    South Moravian Region       1    SG            1
    sixix-2nyqd-t2k2v-vlsyz-dssko-ls4hl-hyij4-y7mdp-ja6cj-nsmpf-yae       1    sg2               1    Posita.si                                       1    Stockholm                   1    SI            1
    sma3p-ivkif-hz7nu-ngmvq-ibnjg-nubke-zf6gh-wbnfc-2dlng-l3die-zqe       1    sh1               1    SyT - Servicios y Telecomunicaciones S.A.       1    Tallinn                     1    US            5
    sqhxa-h6ili-qkwup-ohzwn-yofnm-vvnp5-kxdhg-saabw-rvua3-xp325-zqe       1    ta2               1    TRG                                             1    Tbilisi                     1    ZA       1 -> 2
    ucjqj-jmbj3-rs4aq-ekzpw-ltjs3-zrcma-t6r3t-m5wxc-j5yrj-unwoj-mae       1    tb1               1    Telia DC                                        1    Tel Aviv               1 -> 0
    unqqg-no4b2-vbyad-ytik2-t3vly-3e57q-aje2t-sjb5l-bd4ke-chggn-uqe       1    to1               1    Telin                                           1    Texas                       1
    vegae-c4chr-aetfj-7gzuh-c23sx-u2paz-vmvbn-bcage-pu7lu-mptnn-eqe       1    tv1          1 -> 0    Teraco                                     0 -> 1    Tokyo                       1
    wdjjk-blh44-lxm74-ojj43-rvgf4-j5rie-nm6xs-xvnuv-j3ptn-25t4v-6ae       1    ty1               1    Unicom                                          1    Warszawa                    1
    wdnqm-clqti-im5yf-iapio-avjom-kyppl-xuiza-oaz6z-smmts-52wyg-5ae       1    wa2               1    hkcolo                                     0 -> 1    Zurich                      1
    wwdbq-xuqhf-eydzu-oyl7p-ga565-zm7s7-yrive-ozgsy-zzgh3-qwb3j-cae       1    zh3               1

End result: This selection prevents any of the target topology parameters from going over the limit (beyond those that already are) but worsens the Nakamoto coefficients.

So which is the better measure?

2 Likes

By the way, node xetvj appears to have recovered its health.

And that is exactly the key question. So far in the tooling we have been following the Nakamoto Coefficients across multiple dimensions as the driving factor for the topology since itā€™s the golden standard for measuring network decentralization (e.g. here). For instance, even if we 100% strictly follow the target topology, we still have the NC of 5 for the NNS subnet. We would have the NC of 4 for Country on uzr34 (II subnet). So there is really no need to over-dramatize here or pretend that itā€™s absolutely critical to be super pedantic here. Itā€™s just different measures of decentralization, and all of them are reasonably good. Which one is betterā€¦ I guess depends.
And itā€™s up to us to discuss. In a civilized discussion, of course.

Note that NC is often (on different crypto projects) calculated just in a single dimension ā€“ per ā€œnode providersā€. We calculate it across multiple dimensions, which may or may not be necessary. I personally think itā€™s reasonable and useful to calculate and compare across multiple dimensions. But I donā€™t speak in the name of the entire DFINITY, and Iā€™m not against changing the current implementation if there is a good argument for it.
For instance, one could argue that all City & Country & Continent dimensions arenā€™t as important as the ISP (DC owner) or the DC. And certainly not as important as the NP. One could also argue that they are important. I guess depends on the point of view, and Iā€™d welcome some suggestions on how to do the weighting (importance) of these dimensions on the final NC.

3 Likes

Just to chime in on the discussion here, the IC-network is an operational network on which real businesses are running their (decentralized) applications. So the first priority should be the stability and security of the IC-network and subnets, in particular the key subnets such as NNS, SNS, II and Fiduciary subnets. If 2 or more nodes in a subnet are unhealthy or dead, they should be replaced as soon as possible. When this is not done, and additional nodes become unhealthy or dead in the same subnet - which fortunately has not happened so far - this could stall the subnet or even the IC (if it would be the NNS). The operations team is using the dre-tooling on a daily (operational) basis for subnet monitoring and node replacements, and my opinion is that we should not blur (short term/long term) improvement discussions with operational processes that risk the stability of the IC.

I think everybody agrees that the current DRE tooling uses different coefficients than the target topology, that this needs to be aligned to each other, and that maybe in the future other decentralization coefficients like ISP and Node Operator are relevant to be added. I think everybody also agrees that through all subnet proposals, the goal is to achieve (at some point in the future, preferably earlier than later) the exact target topology that the optimization tooling calculates. Thatā€™s why it is called ā€œtarget topologyā€ and not just ā€œtopologyā€. @timk11 @Lorimer if you are willing to think about and help/contribute creating a roadmap for improvements that would be great.

1 Like

This is a question that has been deliberated and resulted in motion proposals, where the IC Target Topology constraints have been intentionally specified in terms of limits, not in terms of Nakamoto coefficients (which is a related but different concept). This isnā€™t the thread for debating what we should be trying to adhere to (thatā€™s what the motion was for).

Iā€™m not over dramatizing anything. Iā€™m pointing out real problems, and a lack of adherence to the currently standing motion proposal (which has clearly been a pattern).

Iā€™m also pointing out the needlessness of this lack of adherence. This proposal unnecessarily goes against the terms laid out in the motion proposal.

Agreed, this isnā€™t the place for debating what the motion proposal should have contained.

Agreed. I think this proposal should be rejected promptly and replaced with appropriate proposals, which I would be prompt to accept. Thereā€™s no reason this canā€™t be done without taking this subnet even further away from the target topology (which also contributes to loss of security and resilience). A reason has not yet been provided as to why this would not be the best course of action. :slight_smile:


Thanks @SvenF, Iā€™m currently planning a stochastic optimisation tool that should be capable of confirming the achievability of arbitrary target topologies given arbitrary nodes, subnets, characteristics, and constraints. Iā€™m also intending for this to be capable of charting minimal steps (in terms of proposals) to take a topology from an arbitrary state to an optimal state. Iā€™m hoping to have a working prototype soon :slight_smile:

I think the reasons have been very clearly specified, the first one being that the node replacements as calculated by the operational dre tooling give a replacement proposal that optimizes the decentralization coefficients - although it takes into account an extra decentralization parameter that is not used in the target topology, which should be brought in line in the future - and second, rejecting the proposal means submitting a new proposal and another four days of voting which would delay critical node replacements. Operational stability and security of the subnets should be the primary priority. But in the end, it is for the community to decide on the proposal.

1 Like
  1. Thereā€™s no real urgency. One of the two nodes that were down is now back up. In addition, my understanding was that this was part of a planned exercise in order for the NP to run management operations on those nodes.
  2. There are plenty of ways of achieving the desired outcome without taking the subnet further away from the IC target topology.

Given that thereā€™s no urgency, I do not see a justification for proceeding with this proposal (given the currently standing motion that was passed by the community). Voting in favour of this proposal means voting against the terms laid out by that motion (though few that cast their vote will realise this).

The majority of those voting on this proposal have no idea where to look for this sort of critical discussion. The proposal was submitted without a link to a forum topic. I expect it will pass, either because of this, or because the DFINITY known neuron will weigh in (reflecting the opinions that have already been shared by you and Sat).

Thank you @sat and @SvenF for the further discussion on the sorts of priorities that are being applied and that need to be applied here. I absolutely agree that the discussion should stay civilised and respectful, as should be the case for any discussion at all on this forum. You and your teams have been working on these issues full-time for a long time now, whereas Iā€™ve come into this quite recently and embarked upon a huge learning curve to understand whatā€™s involved in keeping the IC running effectively and safely. As part of a team thatā€™s now been tasked with reviewing and voting on Subnet Management, Node Provider and Participant Management proposals (once the new grants for this are formally underway), Iā€™ve found these discussions very helpful for getting my head around all the relevant considerations and developing a way of thinking about these decisions.

My question

was deliberately intended to provoke some further nutting out of the priorities involved. Itā€™s certainly not a black and white question with a single right answer. Part of the question is about how rigorously the target topology should be adhered to, and obviously thereā€™s a balance to be struck. Indeed, there are discrepancies between the current DRE tooling and the target topology, and this will be good to reconcile over time. Thanks again for your very helpful answers to these questions.

4 Likes

Motion proposals should be adhered to where possible - otherwise whatā€™s the point of the motion proposal?

Iā€™d be interested to know why the DRE team are reluctant to swap an appropriate node out of another subnet in order to swap it into this subnet? Is this not the perfect example of when it would be appropriate to do this? If itā€™s not appropriate to do this now, when is it ever appropriate to do this?

Iā€™d also be interested to know why these proposals keep getting submitted without a link to the forum despite multiple requests for this?

Iā€™d also like to say that I think this has been a civilised back a forth (please let me know if this is at odds with anyoneā€™s experience).

A couple of questions to add if thatā€™s okay @sat and @SvenF :

  1. Can I ask where to find information regarding whether a particular node machine has Secure Encrypted Virtualisation (SEV)?
  2. Out of interest, given that subnet membership changes are about updating the registry (while itā€™s the nodeā€™s job to poll the registry and join the appropriate subnet), could a single ā€˜AddNodeToSubnetā€™ proposal be used to reassign a node from one subnet to another (in a single proposal). I can see that historically it hasnā€™t been used this way, but based on my understanding of what the underlying membership mechanism is, it seems like this should be possible.

That is a lot of questions @Lorimer. Let me try to answer them.

This is indeed what we are doing. We are carefully evaluating various aspects though. Every decision has pros and cons and sometimes balancing pros and cons isnā€™t straightforward. Blindly following all previous decisions is not the necessarily the best action in every situation IMHO. But we do give our best to follow adopted motion proposals, and to make course corrections when needed.

We have done such things in the past, and thatā€™s perfectly possible. However, from my experience it may happen that such nodes again move out of the subnet, e.g. when they become unhealthy, and then get added to some other subnets later on. So itā€™s not a silver bullet, especially when there is no support in tooling for this. So rather than doing ad-hoc changes like this, Iā€™d prefer if we make a change in the tooling.
To make this discussion more productive ā€“ any suggestions how to do this?

Simple. We spend a ton of time answering questions in this forum instead of doing actual work. We have a very small team, and we already do a ton of things. Adding a link to the forum post is a fair amount of work and cannot be done quickly because: a) we use dre tool written in rust for submitting these proposals, and I havenā€™t found yet a crate to talk to discourse API from rust, b) adding a link to the forum post, e.g. in the version elect proposals, is a three-step process (1) create a stub forum thread, (2) submit a proposal with a link to the stub forum thread, and (3) update the forum thread/post contents. Obviously can be done but itā€™s work.
Iā€™m now thinking if we could call python from rust (since there is a python client for discourse) or should we implement this client in rust.

In principle yes, although aggressively insisting that others take a particular action without considering their response and without providing sufficient evidence that the particular action must be taken urgently could be seen in other light :wink: To be perfectly honest Iā€™d appreciate a bit more considerate (less ā€œcatastrophicā€ view). But you did have great points, and made an outstanding analysis. So thank you!
As a result of that, we did find an important discrepancy between the dre tooling and the target topology, so I see that as a fantastic outcome. As a result, we will either update the dre tooling or submit another motion proposal, depending on the outcome of additional analysis that we will have to conduct to determine the potential impact of this change. To set expectations right, it will likely take a few weeks before we finish this and agree which changes need to be made. In the meantime weā€™ll likely continue using the tooling as is.

SEV-SNP is not being used right now on the regular IC nodes, due to the subnet recovery challenges. This has been de-prioritized due to the estimated effort needed and unclear benefit (would everyone suddenly jump onto the IC if there was SEV-SNP? ā€“ itā€™s unclear). SEV-SNP will be used on Boundary Nodes, but Iā€™m not sure whatā€™s the timeline for this. The information on whether node supports SEV-SNP will be in the registry but AFAIK this field is not used yet.

In principle yes. We have all the necessary pieces. In practice, itā€™s work. One would have to terminate all activities of the node in the subnet (wait for the next CUP, etc), then after the node is not used in the subnet anymore, prune all data on the node. Then add all data from the new subnet into the node (state sync, takes between a few minutes/hours and a few days, depending on the subnet size and the link speed), and then finally the node becomes an active and productive member.
So yes it can be done in theory. In practice itā€™s something that needs development work. Would you like to help with this, to get your hands dirty?

4 Likes

I just submitted another proposal that more than recovers the subnet decentralization: https://dashboard.internetcomputer.org/proposal/132225

Here is the command I ran (in case someone wonders):

dre subnet replace --id uzr34-akd3s-xrdag-3ql62-ocgoh-ld2ao-tamcv-54e7j-krwgb-2gm4z-oqe -o 4 --motivation ā€œaligning subnet decentralization with the target topologyā€ --forum Subnet Management - uzr34 (II)

I also added support in the DRE tool to manually provide a link to the forum post. Itā€™s not ideal (better would probably be to create a new forum thread automatically), but it was easy. And hopefully it will be sufficient short term.

4 Likes

:heart_eyes:

Thanks Sat :pray:


Proposal 132225

TLDR: Iā€™ve adopted this proposal :heart_on_fire: it brings this subnet inline with the IC target topology! 4 removed nodes replaced with nodes in Israel, South Africa, Croatia and Sri Lanka to improve decentralisation coefficients.

Decentralisation Stats

Subnet node distance stats (distance between any 2 nodes in the subnet) ā†’

Smallest Distance Average Distance Largest Distance
EXISTING 0 km 8050.519 km 19448.574 km
PROPOSED 0 km 7993.733 km (-0.7%) 19448.574 km

This proposal ever so slightly reduces decentralisation considered purely in terms of geographic distance (and therefore thereā€™s a slight theoretical reduction in localised disaster resilience).

Subnet characteristic counts ā†’

Continents Countries Data Centers Owners Node Providers
EXISTING 5 21 28 26 28
PROPOSED 5 23 (+8.7%) 28 28 (+7.1%) 28

This proposal significantly improves decentralisation in terms of jurisdiction diversity and data center owner diversity. :+1:

Largest number of nodes with the same characteristic (e.g. continent, country, data center, etc.) ā†’

Continent Country Data Center Owner Node Provider
EXISTING 10 5 1 2 1
PROPOSED 9 (-10%) 3 (-40%) 1 1 (-50%) 1

See here for acceptable limits ā†’ Motion 132136. After this proposal is executed this subnet will be meeting the IC target topology decentralisation coefficients! :tada:

The above subnet information is illustrated below, followed by a node reference table:

Map Description
  • Red marker represents a removed node (transparent center for overlap visibility)
  • Green marker represents an added node
  • Blue marker represents an unchanged node
  • Highlighted patches represent the country the above nodes sit within (red if the country is removed, green if added, otherwise grey)
  • Light grey markers with yellow borders are examples of unassigned nodes that would be viable candidates for joining the subnet according to formal decentralisation coefficients (so this proposal can be viewed in the context of alternative solutions that are not being used)

Table
Continent Country Data Center Owner Node Provider Node Status
--- Americas Costa Rica San JosƩ 1 (cr1) Navegalo GeoNodes LLC os2wv-dchv6-fjwdg-nichf-3cra6-wccb3-57glq-xqgen-zwobp-aonve-lqe UP
--- Europe Germany Frankfurt 2 (fr2) Equinix Virtual Hive Ltd ysjs7-fe67q-k7pxe-g4puk-d5z7p-xrtg2-xqcmy-pmpoa-7bcfu-pmlqy-rae UP
--- Americas United States of America (the) Atlanta (at1) Flexential BLP22, LLC v6caw-kf4b4-j22f5-pqsvj-hg7dj-wx4xx-y7urh-eixmh-hnvz7-hoe2v-jae UP
--- Americas United States of America (the) Houston (hu1) TRG 43rd Big Idea Films ham46-r7u4j-yoyrk-mrnxu-utzq3-c37he-vcns2-gc5yg-jxyv5-mvlhc-gae UP
+++ Europe Croatia Zagreb 1 (zg1) Anonstake Anonstake q3vac-kcwo2-ruiht-nflb7-ifoev-vkjcw-quybi-ugvgn-pqfwp-jntxi-dqe UNASSIGNED
+++ Asia Israel Tel Aviv 1 (tv1) Interhost GeoNodes LLC rfkza-27bii-6jan4-u4zll-lkvmz-snmao-irmlj-arpdd-kyxrg-xnq3a-7ae UNASSIGNED
+++ Asia Sri Lanka Colombo 1 (cm1) OrionStellar Geodd Pvt Ltd suty3-goyd2-t6ngb-dsgzv-vkvt6-w2x4t-lioxl-2xsaa-ztaly-y2ov2-hae UNASSIGNED
+++ Africa South Africa Cape Town 2 (ct2) Teraco Kontrapunt (Pty) Ltd kgo2t-vidyw-yw2g5-pqwrt-nr227-rbq2o-pog27-zarc2-dfrlw-vvjge-4qe UNASSIGNED
Americas Argentina CABA 1 (ar1) SyT - Servicios y Telecomunicaciones S.A. Mariano Stoll v27at-hedf7-4a2my-tboq6-escdm-77krt-2qfuq-zjptf-z2sbk-vd7zs-xae UP
Oceania Australia Queensland 1 (sc1) NEXTDC ANYPOINT PTY LTD zjiki-pzvnv-m4rnn-fodt3-poon4-uldx7-d3wkq-gptsu-g2mjs-boi35-3qe UP
Europe Belgium Antwerp (an1) Datacenter United Allusion xu2zg-nns7x-z67l6-foa5w-yc4ek-addku-g2hqg-c3jdg-wabdu-5ouaw-yae UP
Americas Canada Fremont (fm1) Hurricane Electric Boolean Bit, LLC z4jw5-v4ee6-aa7gr-5axkc-4ocjy-v5vv5-inwc6-ma4hw-mb7jv-3skxy-eqe UP
Americas Canada Toronto (to1) Cyxtera Blockchain Development Labs x3cey-uerdd-53a7n-d45e2-gjsnd-airg5-nrs5i-4xujk-c5ynl-4pie6-yqe UP
Europe Switzerland Zurich 3 (zh3) Nine.Ch Tomahawk.vc v2pkj-vpsow-fp24q-zqwfj-p3nek-m52xz-oz6ra-blmra-73voj-jvwb5-gae UP
Asia China HongKong 1 (hk1) Unicom Pindar Technology Limited w4ri3-ytfnq-jg3z3-qseka-se4xe-b2fl2-km766-ruzwd-riw72-6bifs-4ae UP
Americas Costa Rica Bogota 1 (bg1) EdgeUno Geeta Kalwani aajth-ndp7x-ro5ok-yikyd-4i7xn-5k5ki-e3d37-hh4gn-s4opz-bnzxf-4qe UP
Europe Czechia South Moravian Region 1 (bn1) Master Internet Maksym Ishchenko zgtrt-4vlgr-pbytl-t2yqq-qf4nk-wyoos-vrfpu-hxqcw-tcnfu-73kjb-pae UP
Europe Spain Madrid 1 (ma1) Ginernet Ivanov Oleksandr yi6r6-u4kax-jphcr-jcqr5-t3zpm-gmp3b-2hiew-iinpf-sgjos-eabha-aqe UP
Europe Estonia Tallinn 2 (ta2) Telia DC Artem Horodyskyi zgeaf-fcq4e-fcnht-g7mpg-sb7ff-r6awk-zvkwp-gkloc-rr6jl-ghsse-mqe UP
Asia Georgia Tbilisi 1 (tb1) Cloud9 George Bassadone uouxk-c246i-dgxzd-ql3a5-koofn-mclrv-toplo-bg76d-l4dzk-ngb3c-nae UP
Asia India New Delhi 1 (nd1) Marvelous Web3 DC Marvelous Web3 67t6p-i4h3c-msv6p-kmbmm-rr6gj-z3nix-d6lo2-mq3q4-3h6rb-lwkbc-lae UP
Asia India Panvel 2 (pl2) Yotta Krishna Enterprises dyycg-wc45f-jwks2-abddo-m3n5r-o5kxy-g5xhm-7ve4u-3tlk7-i7xec-oae UP
Asia Japan Tokyo (ty1) Equinix Starbase go5zz-xs6yg-mylwl-v7uob-7bg4b-wjzhe-vmrwe-uy7mz-ckaz2-idm33-rqe UP
Asia Korea (the Republic of) Seoul 2 (kr2) Gasan Web3game wjwzb-q3ogf-fi3po-kf6y6-wzuuj-3ac3m-kjvab-fufsm-z2skq-kthkx-xae UP
Europe Poland Warszawa 2 (wa2) Central Tower DC Bohatyrov Volodymyr mswad-oq7wj-5r4yy-b5qoy-cmv7z-wzfb3-ktn6l-rcnrz-mni2f-lsys6-wqe UP
Asia Singapore Singapore 2 (sg2) Telin OneSixtyTwo Digital Capital qp3lh-25yxy-dlk4t-ay73d-frr4t-3kmi5-35kqg-3vvbq-26qhh-6xrdr-oqe UP
Europe Slovenia Ljubljana (lj1) Posita.si Fractal Labs AG 6adxp-p7u63-xsdtk-lo6oc-vpqmi-44hgt-yv652-cbm5p-mssge-wsrz6-oqe UP
Europe Sweden Stockholm 1 (sh1) Digital Realty DFINITY Operations SA vgfnl-4phvh-44pk3-yshmp-ckwz3-qnzob-l5wnj-pqn2j-vv5jh-3oewk-xqe UP
Americas United States of America (the) Chicago 3 (ch3) CyrusOne MI Servers gtfa3-saq3t-ymlel-lsf6d-ans7b-cr45x-xg5np-xbxyt-nxfrt-iynyy-5qe UP
Americas United States of America (the) Orlando (or1) Datasite Giant Leaf, LLC z5a4h-43szy-vvp4j-xorii-l6yma-4iyzt-7o3ry-frvqe-azkit-5iag2-rqe UP
Americas United States of America (the) Panama City 1 (pc1) Navegalo Bianca-Martina Rohner y7bml-csbq7-euzyf-njmvm-qfftp-iy7lc-wisaq-jlmul-sdo7p-7lkx4-3ae UP
Africa South Africa Gauteng 2 (jb2) Africa Data Centres Karel Frank xav3a-kdo3a-2rgbg-o6vnk-clat5-bcc7w-vmnej-z55rx-mfx26-7xugo-tqe UP

Known Neurons to follow if you're too busy to keep on top of things like this

If you found this analysis helpful and would like to follow the vote of the LORIMER known neuron in the future, consider configuring LORIMER as a followee for the Subnet Management topic.

Other good neurons to follow:

  • Synapse (Synapse follows the LORIMER and CodeGov known neurons for Subnet Management, and is a generally well informed known neuron to follow on numerous other topics)

  • CodeGov (will soon be committed to actively reviewing and voting on Subnet Management proposals based on those reviews, and is well informed on numerous other technical topics)

  • WaterNeuron (the WaterNeuron DAO frequently discuss proposals like this in order to vote responsibly based on DAO consensus)

4 Likes

Iā€™m working on a proof of concept optimisation approach that Iā€™ll be happy to share shortly :slight_smile:

Thanks @Sat. Itā€™s not been my intention to insist that anything must be done, only that I will reject otherwise. My apologies for how this must come across, but I consider certain principles to be foundational to solid Web3 governance, and I will continue to reject proposals that do not meet those criteria (in particular, stating one thing in a motion proposal, and instead doing another thing in order to save time and effort). Iā€™m happy to be more flexible if an official announcement can be made about the currently standing motion proposal.

Would someone from DFINITY be able to update the latest motion proposal topic to make this clear to all that have displayed an interest in this motion? This would make me feel much more comfortable about potentially adopting proposals that are at odds with the standing motion proposal (if itā€™s officially announced as suspended while further analysis is pending).

Interesting, thanks Sat. Does this mean that the SEV column in the IC target topology also needs disregarding for the time-being? I think this is another thing that could do with making clear in any updated motion.

This is also very interesting. So when new nodes are added to large subnets, such as the NNS, even if their status is UP, they may not actually be participating in consensus for quite a while. Is there any way of detecting when the node has become fully integrated into the subnet, and is contributing productively? Presumably there should be a hard cap on the number of nodes that should be swapped into a subnet at any one time (relative to the failure domain for that subnet). The proposal above swaps 4 in one go. Presumably if it swapped 9+1 nodes then the subnet would stall for an unknown length of time while the new nodes are getting up to speed?

Separate question, just out of interest, if an AddNodeToSubnet proposal were raised today to move a node from one subnet to another (rather than selecting from the unassigned nodes), would this proposal fail or encounter other issues, or would it be expected to work?

Thanks for helping me get up to speed with all these things @Sat, I really appreciate your responses. :slight_smile:

2 Likes

Weā€™ll update either the tooling or the motion proposal forum post (and if needed submit a new motion proposal) as soon as we know what would be the best path forward. So far it seems like everyone agrees that the Continent restriction is not very important, but EU in particular may be. So weā€™re looking in this direction for now. But we are not yet in the state to announce anything.

Correct, for now the SEV column does not have any value. In the future it may, though.

The NNS subnet is actually not large in the amount of data it holds. Iā€™ve asked the public dashboard team to include this information (subnet state size) on the subnet list.

If in a subnet with 3*f + 1 nodes you add+remove more than f nodes, there may be a stall, yes.
The DRE tooling already automatically avoids that in some cases, although we should extend it to more invocations. PRs are welcome! :smiley:

1 Like

New proposal for the subnet:

3 Likes

Proposal 133071

Iā€™ve voted to adopt :+1:

0 removed nodes, and 3 nodes add instead (in South Africa, France, India). As stated in the proposal summary:

Motivation: Increasing the size of the Internet Identity subnet to bring it closer to the target topology

Decentralisation Stats

Subnet node distance stats (distance between any 2 nodes in the subnet) ā†’

Smallest Distance Average Distance Largest Distance
EXISTING 0 km 7993.733 km 19448.574 km
PROPOSED 0 km 7898.216 km (-1.2%) 19448.574 km

This proposal slightly reduces decentralisation, considered purely in terms of geographic distance (and therefore thereā€™s a slight theoretical reduction in localised disaster resilience). :-1:

Subnet characteristic counts ā†’

Continents Countries Data Centers Owners Node Providers
EXISTING 5 23 28 28 28
PROPOSED 5 24 (+4.2%) 31 (+9.7%) 31 (+9.7%) 31 (+9.7%)

This proposal improves decentralisation in terms of subnet characteristic diversity. :+1:

Largest number of nodes with the same characteristic (e.g. continent, country, data center, etc.) ā†’

Continent Country Data Center Owner Node Provider
EXISTING 9 3 1 1 1
PROPOSED 10 (+11.1%) 3 1 1 1

See here for acceptable limits ā†’ Motion 132136

The above subnet information is illustrated below, followed by a node reference table:

Map Description
  • Red marker represents a removed node (transparent center for overlap visibility)
  • Green marker represents an added node
  • Blue marker represents an unchanged node
  • Highlighted patches represent the country the above nodes sit within (red if the country is removed, green if added, otherwise grey)
  • Light grey markers with yellow borders are examples of unassigned nodes that would be viable candidates for joining the subnet according to formal decentralisation coefficients (so this proposal can be viewed in the context of alternative solutions that are not being used)

Table
Continent Country Data Center Owner Node Provider Node Status
+++ Europe France Paris 1 (pr1) Celeste Carbon Twelve atjbz-kcjz7-y4mgn-t5wqp-3emfk-6mtlx-ln5i7-4pixf-ocjgh-hfu77-bqe UNASSIGNED
+++ Asia India Navi Mumbai 1 (nm1) Rivram Rivram Inc ecxbl-3dp33-mpskv-yvs6f-674ct-tpr4d-dhzdg-jfgf4-gzhny-n6zzx-lqe UNASSIGNED
+++ Africa South Africa Gauteng 3 (jb3) Xneelo Wolkboer (Pty) Ltd kwryq-ezysk-c4ono-aet7a-hh6h5-4o3bb-a33et-ef4g5-42tot-zaek6-fae UNASSIGNED
Americas Argentina CABA 1 (ar1) SyT - Servicios y Telecomunicaciones S.A. Mariano Stoll v27at-hedf7-4a2my-tboq6-escdm-77krt-2qfuq-zjptf-z2sbk-vd7zs-xae UP
Oceania Australia Queensland 1 (sc1) NEXTDC ANYPOINT PTY LTD zjiki-pzvnv-m4rnn-fodt3-poon4-uldx7-d3wkq-gptsu-g2mjs-boi35-3qe UP
Europe Belgium Antwerp (an1) Datacenter United Allusion xu2zg-nns7x-z67l6-foa5w-yc4ek-addku-g2hqg-c3jdg-wabdu-5ouaw-yae UP
Americas Canada Fremont (fm1) Hurricane Electric Boolean Bit, LLC z4jw5-v4ee6-aa7gr-5axkc-4ocjy-v5vv5-inwc6-ma4hw-mb7jv-3skxy-eqe UP
Americas Canada Toronto (to1) Cyxtera Blockchain Development Labs x3cey-uerdd-53a7n-d45e2-gjsnd-airg5-nrs5i-4xujk-c5ynl-4pie6-yqe UP
Europe Switzerland Zurich 3 (zh3) Nine.Ch Tomahawk.vc v2pkj-vpsow-fp24q-zqwfj-p3nek-m52xz-oz6ra-blmra-73voj-jvwb5-gae UP
Asia China HongKong 1 (hk1) Unicom Pindar Technology Limited w4ri3-ytfnq-jg3z3-qseka-se4xe-b2fl2-km766-ruzwd-riw72-6bifs-4ae UP
Americas Costa Rica Bogota 1 (bg1) EdgeUno Geeta Kalwani aajth-ndp7x-ro5ok-yikyd-4i7xn-5k5ki-e3d37-hh4gn-s4opz-bnzxf-4qe UP
Europe Czechia South Moravian Region 1 (bn1) Master Internet Maksym Ishchenko zgtrt-4vlgr-pbytl-t2yqq-qf4nk-wyoos-vrfpu-hxqcw-tcnfu-73kjb-pae UP
Europe Spain Madrid 1 (ma1) Ginernet Ivanov Oleksandr yi6r6-u4kax-jphcr-jcqr5-t3zpm-gmp3b-2hiew-iinpf-sgjos-eabha-aqe UP
Europe Estonia Tallinn 2 (ta2) Telia DC Artem Horodyskyi zgeaf-fcq4e-fcnht-g7mpg-sb7ff-r6awk-zvkwp-gkloc-rr6jl-ghsse-mqe UP
Asia Georgia Tbilisi 1 (tb1) Cloud9 George Bassadone uouxk-c246i-dgxzd-ql3a5-koofn-mclrv-toplo-bg76d-l4dzk-ngb3c-nae UP
Europe Croatia Zagreb 1 (zg1) Anonstake Anonstake q3vac-kcwo2-ruiht-nflb7-ifoev-vkjcw-quybi-ugvgn-pqfwp-jntxi-dqe UP
Asia India New Delhi 1 (nd1) Marvelous Web3 DC Marvelous Web3 67t6p-i4h3c-msv6p-kmbmm-rr6gj-z3nix-d6lo2-mq3q4-3h6rb-lwkbc-lae UP
Asia India Panvel 2 (pl2) Yotta Krishna Enterprises dyycg-wc45f-jwks2-abddo-m3n5r-o5kxy-g5xhm-7ve4u-3tlk7-i7xec-oae UP
Asia Israel Tel Aviv 1 (tv1) Interhost GeoNodes LLC rfkza-27bii-6jan4-u4zll-lkvmz-snmao-irmlj-arpdd-kyxrg-xnq3a-7ae UP
Asia Japan Tokyo (ty1) Equinix Starbase go5zz-xs6yg-mylwl-v7uob-7bg4b-wjzhe-vmrwe-uy7mz-ckaz2-idm33-rqe UP
Asia Korea (the Republic of) Seoul 2 (kr2) Gasan Web3game wjwzb-q3ogf-fi3po-kf6y6-wzuuj-3ac3m-kjvab-fufsm-z2skq-kthkx-xae UP
Asia Sri Lanka Colombo 1 (cm1) OrionStellar Geodd Pvt Ltd suty3-goyd2-t6ngb-dsgzv-vkvt6-w2x4t-lioxl-2xsaa-ztaly-y2ov2-hae UP
Europe Poland Warszawa 2 (wa2) Central Tower DC Bohatyrov Volodymyr mswad-oq7wj-5r4yy-b5qoy-cmv7z-wzfb3-ktn6l-rcnrz-mni2f-lsys6-wqe UP
Asia Singapore Singapore 2 (sg2) Telin OneSixtyTwo Digital Capital qp3lh-25yxy-dlk4t-ay73d-frr4t-3kmi5-35kqg-3vvbq-26qhh-6xrdr-oqe UP
Europe Slovenia Ljubljana (lj1) Posita.si Fractal Labs AG 6adxp-p7u63-xsdtk-lo6oc-vpqmi-44hgt-yv652-cbm5p-mssge-wsrz6-oqe UP
Europe Sweden Stockholm 1 (sh1) Digital Realty DFINITY Operations SA vgfnl-4phvh-44pk3-yshmp-ckwz3-qnzob-l5wnj-pqn2j-vv5jh-3oewk-xqe UP
Americas United States of America (the) Chicago 3 (ch3) CyrusOne MI Servers gtfa3-saq3t-ymlel-lsf6d-ans7b-cr45x-xg5np-xbxyt-nxfrt-iynyy-5qe UP
Americas United States of America (the) Orlando (or1) Datasite Giant Leaf, LLC z5a4h-43szy-vvp4j-xorii-l6yma-4iyzt-7o3ry-frvqe-azkit-5iag2-rqe UP
Americas United States of America (the) Panama City 1 (pc1) Navegalo Bianca-Martina Rohner y7bml-csbq7-euzyf-njmvm-qfftp-iy7lc-wisaq-jlmul-sdo7p-7lkx4-3ae UP
Africa South Africa Cape Town 2 (ct2) Teraco Kontrapunt (Pty) Ltd kgo2t-vidyw-yw2g5-pqwrt-nr227-rbq2o-pog27-zarc2-dfrlw-vvjge-4qe UP
Africa South Africa Gauteng 2 (jb2) Africa Data Centres Karel Frank xav3a-kdo3a-2rgbg-o6vnk-clat5-bcc7w-vmnej-z55rx-mfx26-7xugo-tqe UP

Known Neurons to follow if you're too busy to keep on top of things like this

If you found this analysis helpful and would like to follow the vote of the LORIMER known neuron in the future, consider configuring LORIMER as a followee for the Subnet Management topic.

Other good neurons to follow:

  • Synapse (follows the LORIMER and CodeGov known neurons for Subnet Management, and is a generally well informed known neuron to follow on numerous other topics)

  • CodeGov (actively reviews and votes on Subnet Management proposals, and is well informed on numerous other technical topics)

  • WaterNeuron (the WaterNeuron DAO frequently discuss proposals like this in order to vote responsibly based on DAO consensus)

2 Likes

Voted to adopt the addition of the three healthy nodes that increases the size of the subnet while the decentralization change is over all slightly better.

2 Likes