I tried a somewhat cruder approach of using the decentralization tool to tabulate all the network nodes and then re-sorting the table by the various parameters in order to search for eligible replacement nodes. I came up with a number of possibilities, but how about these ones for example?
Overall replacement impact: the average log2 of Nakamoto Coefficients across all features changes from 2.7868 to 2.6640
Details
Nodes removed:
ā xetvj-ysqpy-wnnd4-fbytw-n6arq-w7f5e-fhsuo-7xd67-lwpt2-novnx-iae [health: healthy, impact on decentralization: the number of different NP actors changes from 28 to 27]
ā ylm4h-mc2vy-vmwvf-g2qhn-36rbw-5cunq-wpj2j-ttofo-xc27c-vfdh7-aae [health: dead, impact on decentralization: the minimum Nakamoto coefficient across all features changes from 2 to 1]
Nodes added:
++> 7fiyj-xjxkj-fjvbl-tl4rp-6xvg3-moow7-dwtyi-nsxeg-unw7n-ivshf-pqe [health: healthy, impact on decentralization: the minimum Nakamoto coefficient across all features changes from 1 to 2]
++> hlc73-cklhu-zo4mt-waxb6-ew5tx-gm4or-coy2j-xnwmy-xsycn-j6f4e-iae [health: healthy, impact on decentralization: the average log2 of Nakamoto Coefficients across all features changes from 2.7429 to 2.6640]
node_provider data_center data_center_owner city country continent
------------- ----------- ----------------- ---- ------- ---------
4fedi-eu6ue-nd7ts-vnof5-hzg66-hgzl7-liy5n-3otyp-h7ipw-owycg-uae 0 -> 1 an1 1 Africa Data Centres 1 Bogota 1 AR 1 Africa 1 -> 2
4r6qy-tljxg-slziw-zoteo-pboxh-vlctz-hkv2d-7zior-u3pxm-mmuxb-cae 1 ar1 1 Central Tower DC 1 CABA 1 AU 1 Asia 8 -> 7
64xe5-tx2s3-4gjmj-pnozr-fejw2-77y5y-rhcjk-glnmx-62brf-qin5q-pqe 1 at1 1 Cloud9 1 California 1 BE 1 Europe 9
6nbcy-kprg6-ax3db-kh3cz-7jllk-oceyh-jznhs-riguq-fvk6z-6tsds-rqe 1 bg1 1 CyrusOne 1 Cape Town 0 -> 1 CA 1 North America 7
6sq7t-knkul-fko6h-xzvnf-ktbvr-jhx7r-hapzr-kjlek-whugy-zt6ip-xqe 1 -> 0 bn1 1 Cyxtera 1 Colombo 1 -> 0 CH 1 Oceania 1
7at4h-nhtvt-a4s55-jigss-wr2ha-ysxkn-e6w7x-7ggnm-qd3d5-ry66r-cae 1 ch3 1 Datacenter United 1 Flanders 1 CO 1 South America 2
7uioy-xitfw-yqcko-5gpya-3lpsw-dw7zt-dyyyf-wfqif-jvi76-fdbkg-cqe 1 cm1 1 -> 0 Datasite 1 Florida 1 CZ 1
bvcsg-3od6r-jnydw-eysln-aql7w-td5zn-ay5m6-sibd2-jzojt-anwag-mqe 1 ct2 0 -> 1 Digital Realty 1 Gauteng 1 DE 1
dhywe-eouw6-hstpj-ahsnw-xnjxq-cmqks-47mrg-nnncb-3sr5d-rac6m-nae 1 fm1 1 EdgeUno 1 Georgia 1 EE 1
diyay-s4rfq-xnx23-zczwi-nptra-5254n-e4zn6-p7tqe-vqhzr-sd4gd-bqe 1 fr2 1 Equinix 2 Hesse 1 ES 1
eatbv-nlydd-n655c-g7j7p-gnmpz-pszdg-6e6et-veobv-ftz2y-4m752-vqe 1 hk1 1 Flexential 1 HongKong 1 -> 2 GE 1
eybf4-6t6bb-unfb2-h2hhn-rrfi2-cd2vs-phksn-jdmbn-i463m-4lzds-vqe 1 -> 0 hk3 0 -> 1 Gasan 1 Illinois 1 HK 1 -> 2
fwnmn-zn7yt-5jaia-fkxlr-dzwyu-keguq-npfxq-mc72w-exeae-n5thj-oae 1 hu1 1 Ginernet 1 Ljubljana 1 IL 1 -> 0
ivf2y-crxj4-y6ewo-un35q-a7pum-wqmbw-pkepy-d6uew-bfmff-g5yxe-eae 1 jb2 1 Hurricane Electric 1 Madrid 1 IN 1
izmhk-lpjum-uo4oy-lviba-yctpc-arg4b-2ywim-vgoiu-gqaj2-gskmw-2qe 1 kr2 1 Interhost 1 -> 0 New Delhi 1 JP 1
otzuu-dldzs-avvu2-qwowd-hdj73-aocy7-lacgi-carzj-m6f2r-ffluy-fae 1 lj1 1 Marvelous Web3 DC 1 Ontario 1 KR 1
py2kr-ipr2p-ryh66-x3a3v-5ts6u-7rfhf-alkna-ueffh-hz5ox-lt6du-qqe 0 -> 1 ma1 1 Master Internet 1 Panama City 1 LK 1 -> 0
qdj4d-76lh3-w2q5i-kwjcd-643pq-pk42d-cziag-4hkau-35gib-m7s33-6qe 1 nd1 1 NEXTDC 1 Queensland 1 PA 1
r3yjn-kthmg-pfgmb-2fngg-5c7d7-t6kqg-wi37r-j7gy6-iee64-kjdja-jae 1 or1 1 Navegalo 1 Seoul 1 PL 1
rbn2y-6vfsb-gv35j-4cyvy-pzbdu-e5aum-jzjg6-5b4n5-vuguf-ycubq-zae 1 pc1 1 Nine.Ch 1 Singapore 1 SE 1
s5nvr-ipdxf-xg6wd-ofacm-7tl4i-nwjzx-uulum-cugwb-kbpsa-wrsgs-cae 1 sc1 1 OrionStellar 1 -> 0 South Moravian Region 1 SG 1
sixix-2nyqd-t2k2v-vlsyz-dssko-ls4hl-hyij4-y7mdp-ja6cj-nsmpf-yae 1 sg2 1 Posita.si 1 Stockholm 1 SI 1
sma3p-ivkif-hz7nu-ngmvq-ibnjg-nubke-zf6gh-wbnfc-2dlng-l3die-zqe 1 sh1 1 SyT - Servicios y Telecomunicaciones S.A. 1 Tallinn 1 US 5
sqhxa-h6ili-qkwup-ohzwn-yofnm-vvnp5-kxdhg-saabw-rvua3-xp325-zqe 1 ta2 1 TRG 1 Tbilisi 1 ZA 1 -> 2
ucjqj-jmbj3-rs4aq-ekzpw-ltjs3-zrcma-t6r3t-m5wxc-j5yrj-unwoj-mae 1 tb1 1 Telia DC 1 Tel Aviv 1 -> 0
unqqg-no4b2-vbyad-ytik2-t3vly-3e57q-aje2t-sjb5l-bd4ke-chggn-uqe 1 to1 1 Telin 1 Texas 1
vegae-c4chr-aetfj-7gzuh-c23sx-u2paz-vmvbn-bcage-pu7lu-mptnn-eqe 1 tv1 1 -> 0 Teraco 0 -> 1 Tokyo 1
wdjjk-blh44-lxm74-ojj43-rvgf4-j5rie-nm6xs-xvnuv-j3ptn-25t4v-6ae 1 ty1 1 Unicom 1 Warszawa 1
wdnqm-clqti-im5yf-iapio-avjom-kyppl-xuiza-oaz6z-smmts-52wyg-5ae 1 wa2 1 hkcolo 0 -> 1 Zurich 1
wwdbq-xuqhf-eydzu-oyl7p-ga565-zm7s7-yrive-ozgsy-zzgh3-qwb3j-cae 1 zh3 1
End result: This selection prevents any of the target topology parameters from going over the limit (beyond those that already are) but worsens the Nakamoto coefficients.
And that is exactly the key question. So far in the tooling we have been following the Nakamoto Coefficients across multiple dimensions as the driving factor for the topology since itās the golden standard for measuring network decentralization (e.g. here). For instance, even if we 100% strictly follow the target topology, we still have the NC of 5 for the NNS subnet. We would have the NC of 4 for Country on uzr34 (II subnet). So there is really no need to over-dramatize here or pretend that itās absolutely critical to be super pedantic here. Itās just different measures of decentralization, and all of them are reasonably good. Which one is betterā¦ I guess depends.
And itās up to us to discuss. In a civilized discussion, of course.
Note that NC is often (on different crypto projects) calculated just in a single dimension ā per ānode providersā. We calculate it across multiple dimensions, which may or may not be necessary. I personally think itās reasonable and useful to calculate and compare across multiple dimensions. But I donāt speak in the name of the entire DFINITY, and Iām not against changing the current implementation if there is a good argument for it.
For instance, one could argue that all City & Country & Continent dimensions arenāt as important as the ISP (DC owner) or the DC. And certainly not as important as the NP. One could also argue that they are important. I guess depends on the point of view, and Iād welcome some suggestions on how to do the weighting (importance) of these dimensions on the final NC.
Just to chime in on the discussion here, the IC-network is an operational network on which real businesses are running their (decentralized) applications. So the first priority should be the stability and security of the IC-network and subnets, in particular the key subnets such as NNS, SNS, II and Fiduciary subnets. If 2 or more nodes in a subnet are unhealthy or dead, they should be replaced as soon as possible. When this is not done, and additional nodes become unhealthy or dead in the same subnet - which fortunately has not happened so far - this could stall the subnet or even the IC (if it would be the NNS). The operations team is using the dre-tooling on a daily (operational) basis for subnet monitoring and node replacements, and my opinion is that we should not blur (short term/long term) improvement discussions with operational processes that risk the stability of the IC.
I think everybody agrees that the current DRE tooling uses different coefficients than the target topology, that this needs to be aligned to each other, and that maybe in the future other decentralization coefficients like ISP and Node Operator are relevant to be added. I think everybody also agrees that through all subnet proposals, the goal is to achieve (at some point in the future, preferably earlier than later) the exact target topology that the optimization tooling calculates. Thatās why it is called ātarget topologyā and not just ātopologyā. @timk11@Lorimer if you are willing to think about and help/contribute creating a roadmap for improvements that would be great.
This is a question that has been deliberated and resulted in motion proposals, where the IC Target Topology constraints have been intentionally specified in terms of limits, not in terms of Nakamoto coefficients (which is a related but different concept). This isnāt the thread for debating what we should be trying to adhere to (thatās what the motion was for).
Iām not over dramatizing anything. Iām pointing out real problems, and a lack of adherence to the currently standing motion proposal (which has clearly been a pattern).
Iām also pointing out the needlessness of this lack of adherence. This proposal unnecessarily goes against the terms laid out in the motion proposal.
Agreed, this isnāt the place for debating what the motion proposal should have contained.
Agreed. I think this proposal should be rejected promptly and replaced with appropriate proposals, which I would be prompt to accept. Thereās no reason this canāt be done without taking this subnet even further away from the target topology (which also contributes to loss of security and resilience). A reason has not yet been provided as to why this would not be the best course of action.
Thanks @SvenF, Iām currently planning a stochastic optimisation tool that should be capable of confirming the achievability of arbitrary target topologies given arbitrary nodes, subnets, characteristics, and constraints. Iām also intending for this to be capable of charting minimal steps (in terms of proposals) to take a topology from an arbitrary state to an optimal state. Iām hoping to have a working prototype soon
I think the reasons have been very clearly specified, the first one being that the node replacements as calculated by the operational dre tooling give a replacement proposal that optimizes the decentralization coefficients - although it takes into account an extra decentralization parameter that is not used in the target topology, which should be brought in line in the future - and second, rejecting the proposal means submitting a new proposal and another four days of voting which would delay critical node replacements. Operational stability and security of the subnets should be the primary priority. But in the end, it is for the community to decide on the proposal.
Thereās no real urgency. One of the two nodes that were down is now back up. In addition, my understanding was that this was part of a planned exercise in order for the NP to run management operations on those nodes.
There are plenty of ways of achieving the desired outcome without taking the subnet further away from the IC target topology.
Given that thereās no urgency, I do not see a justification for proceeding with this proposal (given the currently standing motion that was passed by the community). Voting in favour of this proposal means voting against the terms laid out by that motion (though few that cast their vote will realise this).
The majority of those voting on this proposal have no idea where to look for this sort of critical discussion. The proposal was submitted without a link to a forum topic. I expect it will pass, either because of this, or because the DFINITY known neuron will weigh in (reflecting the opinions that have already been shared by you and Sat).
Thank you @sat and @SvenF for the further discussion on the sorts of priorities that are being applied and that need to be applied here. I absolutely agree that the discussion should stay civilised and respectful, as should be the case for any discussion at all on this forum. You and your teams have been working on these issues full-time for a long time now, whereas Iāve come into this quite recently and embarked upon a huge learning curve to understand whatās involved in keeping the IC running effectively and safely. As part of a team thatās now been tasked with reviewing and voting on Subnet Management, Node Provider and Participant Management proposals (once the new grants for this are formally underway), Iāve found these discussions very helpful for getting my head around all the relevant considerations and developing a way of thinking about these decisions.
My question
was deliberately intended to provoke some further nutting out of the priorities involved. Itās certainly not a black and white question with a single right answer. Part of the question is about how rigorously the target topology should be adhered to, and obviously thereās a balance to be struck. Indeed, there are discrepancies between the current DRE tooling and the target topology, and this will be good to reconcile over time. Thanks again for your very helpful answers to these questions.
Motion proposals should be adhered to where possible - otherwise whatās the point of the motion proposal?
Iād be interested to know why the DRE team are reluctant to swap an appropriate node out of another subnet in order to swap it into this subnet? Is this not the perfect example of when it would be appropriate to do this? If itās not appropriate to do this now, when is it ever appropriate to do this?
Iād also be interested to know why these proposals keep getting submitted without a link to the forum despite multiple requests for this?
Iād also like to say that I think this has been a civilised back a forth (please let me know if this is at odds with anyoneās experience).
A couple of questions to add if thatās okay @sat and @SvenF :
Can I ask where to find information regarding whether a particular node machine has Secure Encrypted Virtualisation (SEV)?
Out of interest, given that subnet membership changes are about updating the registry (while itās the nodeās job to poll the registry and join the appropriate subnet), could a single āAddNodeToSubnetā proposal be used to reassign a node from one subnet to another (in a single proposal). I can see that historically it hasnāt been used this way, but based on my understanding of what the underlying membership mechanism is, it seems like this should be possible.
That is a lot of questions @Lorimer. Let me try to answer them.
This is indeed what we are doing. We are carefully evaluating various aspects though. Every decision has pros and cons and sometimes balancing pros and cons isnāt straightforward. Blindly following all previous decisions is not the necessarily the best action in every situation IMHO. But we do give our best to follow adopted motion proposals, and to make course corrections when needed.
We have done such things in the past, and thatās perfectly possible. However, from my experience it may happen that such nodes again move out of the subnet, e.g. when they become unhealthy, and then get added to some other subnets later on. So itās not a silver bullet, especially when there is no support in tooling for this. So rather than doing ad-hoc changes like this, Iād prefer if we make a change in the tooling.
To make this discussion more productive ā any suggestions how to do this?
Simple. We spend a ton of time answering questions in this forum instead of doing actual work. We have a very small team, and we already do a ton of things. Adding a link to the forum post is a fair amount of work and cannot be done quickly because: a) we use dre tool written in rust for submitting these proposals, and I havenāt found yet a crate to talk to discourse API from rust, b) adding a link to the forum post, e.g. in the version elect proposals, is a three-step process (1) create a stub forum thread, (2) submit a proposal with a link to the stub forum thread, and (3) update the forum thread/post contents. Obviously can be done but itās work.
Iām now thinking if we could call python from rust (since there is a python client for discourse) or should we implement this client in rust.
In principle yes, although aggressively insisting that others take a particular action without considering their response and without providing sufficient evidence that the particular action must be taken urgently could be seen in other light To be perfectly honest Iād appreciate a bit more considerate (less ācatastrophicā view). But you did have great points, and made an outstanding analysis. So thank you!
As a result of that, we did find an important discrepancy between the dre tooling and the target topology, so I see that as a fantastic outcome. As a result, we will either update the dre tooling or submit another motion proposal, depending on the outcome of additional analysis that we will have to conduct to determine the potential impact of this change. To set expectations right, it will likely take a few weeks before we finish this and agree which changes need to be made. In the meantime weāll likely continue using the tooling as is.
SEV-SNP is not being used right now on the regular IC nodes, due to the subnet recovery challenges. This has been de-prioritized due to the estimated effort needed and unclear benefit (would everyone suddenly jump onto the IC if there was SEV-SNP? ā itās unclear). SEV-SNP will be used on Boundary Nodes, but Iām not sure whatās the timeline for this. The information on whether node supports SEV-SNP will be in the registry but AFAIK this field is not used yet.
In principle yes. We have all the necessary pieces. In practice, itās work. One would have to terminate all activities of the node in the subnet (wait for the next CUP, etc), then after the node is not used in the subnet anymore, prune all data on the node. Then add all data from the new subnet into the node (state sync, takes between a few minutes/hours and a few days, depending on the subnet size and the link speed), and then finally the node becomes an active and productive member.
So yes it can be done in theory. In practice itās something that needs development work. Would you like to help with this, to get your hands dirty?
Here is the command I ran (in case someone wonders):
dre subnet replace --id uzr34-akd3s-xrdag-3ql62-ocgoh-ld2ao-tamcv-54e7j-krwgb-2gm4z-oqe -o 4 --motivation āaligning subnet decentralization with the target topologyā --forum Subnet Management - uzr34 (II)
I also added support in the DRE tool to manually provide a link to the forum post. Itās not ideal (better would probably be to create a new forum thread automatically), but it was easy. And hopefully it will be sufficient short term.
TLDR: Iāve adopted this proposal it brings this subnet inline with the IC target topology! 4 removed nodes replaced with nodes in Israel, South Africa, Croatia and Sri Lanka to improve decentralisation coefficients.
Decentralisation Stats
Subnet node distance stats (distance between any 2 nodes in the subnet) ā
Smallest Distance
Average Distance
Largest Distance
EXISTING
0 km
8050.519 km
19448.574 km
PROPOSED
0 km
7993.733 km (-0.7%)
19448.574 km
This proposal ever so slightly reduces decentralisation considered purely in terms of geographic distance (and therefore thereās a slight theoretical reduction in localised disaster resilience).
Subnet characteristic counts ā
Continents
Countries
Data Centers
Owners
Node Providers
EXISTING
5
21
28
26
28
PROPOSED
5
23 (+8.7%)
28
28 (+7.1%)
28
This proposal significantly improves decentralisation in terms of jurisdiction diversity and data center owner diversity.
Largest number of nodes with the same characteristic (e.g. continent, country, data center, etc.) ā
Continent
Country
Data Center
Owner
Node Provider
EXISTING
10
5
1
2
1
PROPOSED
9 (-10%)
3 (-40%)
1
1 (-50%)
1
See here for acceptable limits ā Motion 132136. After this proposal is executed this subnet will be meeting the IC target topology decentralisation coefficients!
The above subnet information is illustrated below, followed by a node reference table:
Map Description
Red marker represents a removed node (transparent center for overlap visibility)
Green marker represents an added node
Blue marker represents an unchanged node
Highlighted patches represent the country the above nodes sit within (red if the country is removed, green if added, otherwise grey)
Light grey markers with yellow borders are examples of unassigned nodes that would be viable candidates for joining the subnet according to formal decentralisation coefficients (so this proposal can be viewed in the context of alternative solutions that are not being used)
Known Neurons to follow if you're too busy to keep on top of things like this
If you found this analysis helpful and would like to follow the vote of the LORIMER known neuron in the future, consider configuring LORIMER as a followee for the Subnet Management topic.
Other good neurons to follow:
Synapse (Synapse follows the LORIMER and CodeGov known neurons for Subnet Management, and is a generally well informed known neuron to follow on numerous other topics)
CodeGov (will soon be committed to actively reviewing and voting on Subnet Management proposals based on those reviews, and is well informed on numerous other technical topics)
WaterNeuron (the WaterNeuron DAO frequently discuss proposals like this in order to vote responsibly based on DAO consensus)
Iām working on a proof of concept optimisation approach that Iāll be happy to share shortly
Thanks @Sat. Itās not been my intention to insist that anything must be done, only that I will reject otherwise. My apologies for how this must come across, but I consider certain principles to be foundational to solid Web3 governance, and I will continue to reject proposals that do not meet those criteria (in particular, stating one thing in a motion proposal, and instead doing another thing in order to save time and effort). Iām happy to be more flexible if an official announcement can be made about the currently standing motion proposal.
Would someone from DFINITY be able to update the latest motion proposal topic to make this clear to all that have displayed an interest in this motion? This would make me feel much more comfortable about potentially adopting proposals that are at odds with the standing motion proposal (if itās officially announced as suspended while further analysis is pending).
Interesting, thanks Sat. Does this mean that the SEV column in the IC target topology also needs disregarding for the time-being? I think this is another thing that could do with making clear in any updated motion.
This is also very interesting. So when new nodes are added to large subnets, such as the NNS, even if their status is UP, they may not actually be participating in consensus for quite a while. Is there any way of detecting when the node has become fully integrated into the subnet, and is contributing productively? Presumably there should be a hard cap on the number of nodes that should be swapped into a subnet at any one time (relative to the failure domain for that subnet). The proposal above swaps 4 in one go. Presumably if it swapped 9+1 nodes then the subnet would stall for an unknown length of time while the new nodes are getting up to speed?
Separate question, just out of interest, if an AddNodeToSubnet proposal were raised today to move a node from one subnet to another (rather than selecting from the unassigned nodes), would this proposal fail or encounter other issues, or would it be expected to work?
Thanks for helping me get up to speed with all these things @Sat, I really appreciate your responses.
Weāll update either the tooling or the motion proposal forum post (and if needed submit a new motion proposal) as soon as we know what would be the best path forward. So far it seems like everyone agrees that the Continent restriction is not very important, but EU in particular may be. So weāre looking in this direction for now. But we are not yet in the state to announce anything.
Correct, for now the SEV column does not have any value. In the future it may, though.
The NNS subnet is actually not large in the amount of data it holds. Iāve asked the public dashboard team to include this information (subnet state size) on the subnet list.
If in a subnet with 3*f + 1 nodes you add+remove more than f nodes, there may be a stall, yes.
The DRE tooling already automatically avoids that in some cases, although we should extend it to more invocations. PRs are welcome!
0 removed nodes, and 3 nodes add instead (in South Africa, France, India). As stated in the proposal summary:
Motivation: Increasing the size of the Internet Identity subnet to bring it closer to the target topology
Decentralisation Stats
Subnet node distance stats (distance between any 2 nodes in the subnet) ā
Smallest Distance
Average Distance
Largest Distance
EXISTING
0 km
7993.733 km
19448.574 km
PROPOSED
0 km
7898.216 km (-1.2%)
19448.574 km
This proposal slightly reduces decentralisation, considered purely in terms of geographic distance (and therefore thereās a slight theoretical reduction in localised disaster resilience).
Subnet characteristic counts ā
Continents
Countries
Data Centers
Owners
Node Providers
EXISTING
5
23
28
28
28
PROPOSED
5
24 (+4.2%)
31 (+9.7%)
31 (+9.7%)
31 (+9.7%)
This proposal improves decentralisation in terms of subnet characteristic diversity.
Largest number of nodes with the same characteristic (e.g. continent, country, data center, etc.) ā
The above subnet information is illustrated below, followed by a node reference table:
Map Description
Red marker represents a removed node (transparent center for overlap visibility)
Green marker represents an added node
Blue marker represents an unchanged node
Highlighted patches represent the country the above nodes sit within (red if the country is removed, green if added, otherwise grey)
Light grey markers with yellow borders are examples of unassigned nodes that would be viable candidates for joining the subnet according to formal decentralisation coefficients (so this proposal can be viewed in the context of alternative solutions that are not being used)
Known Neurons to follow if you're too busy to keep on top of things like this
If you found this analysis helpful and would like to follow the vote of the LORIMER known neuron in the future, consider configuring LORIMER as a followee for the Subnet Management topic.
Other good neurons to follow:
Synapse (follows the LORIMER and CodeGov known neurons for Subnet Management, and is a generally well informed known neuron to follow on numerous other topics)
CodeGov (actively reviews and votes on Subnet Management proposals, and is well informed on numerous other technical topics)
WaterNeuron (the WaterNeuron DAO frequently discuss proposals like this in order to vote responsibly based on DAO consensus)
Voted to adopt the addition of the three healthy nodes that increases the size of the subnet while the decentralization change is over all slightly better.