This topic is intended to capture Subnet Management activities over time for the 4zbus subnet, providing a place to ask questions and make observations about the management of this subnet.
At the time of creating this topic the current subnet configuration is as follows:
The other node that’s being replaced appears healthy, and the described rationale is to improve subnet decentralisation. I don’t think it’s clear how this is achieved by the proposed node change. There are currently 4 nodes located in the US, and 6 nodes in Europe. This proposal removes one of the US nodes and replaces it with a node located in Germany (granted this is a new country for the subnet, but it’s yet another European node…).
@andrea or @bitdivine are you the guys to ask for this? Are you able to offer any clarity regarding the ‘improve subnet decentralisation’ element of this proposal?
I think when nodes are replaced, the proposals aims to improve on various decentralization metrics for the subnet, e.g. number of node providers, continents, jurisdictions, etc. Sometimes replacing multiple nodes makes it easier to improve some metrics, than replacing a single node. I don’t actually know for this specific case, let me ping somebody that could share more details.
I think you’re right, it was a great suggestion! This particular proposal was also interesting because a removed node was in the same location as an added node. Hopefully the other changes I made make this sort of situation clear (let me know if you have any other suggestions )
^ This proposal was rejected (DFINITY noticed that one of the proposed nodes was due to be decommissioned - the proposal for that decommissioning is now out → Proposal: 131758).
There’s now a revised open proposal for this subnet → Proposal 131789. Thanks again @sat, the proposal summary is now super informative!
I’ve independently verified the proposal claims below. As before, one node is down and is replaced by an up node by the same node provider in Japan. The other node change is intended to optimise subnet decentralisation coefficients. This optimisation appears to be mostly positive.
Subnet node distance stats (distance between any 2 nodes in the subnet) →
Smallest Distance
Average Distance
Largest Distance
EXISTING
71.122 km
6513.222 km
16469.974 km
PROPOSED
71.122 km
6215.817 km (-4.6%)
16469.974 km
This proposal slightly reduces decentralisation considered purely in terms of geographic distance (and therefore there’s a slight theoretical reduction in localised disaster resilience).
Subnet characteristic counts →
Continents
Countries
Data Centers
Owners
Node Providers
EXISTING
3
9
13
13
13
PROPOSED
3
10 (+10%)
13
13
13
This proposal slightly increases the number of countries that constitute this subnet, improving decentralisation in terms of jurisdiction diversity.
Largest number of nodes with the same characteristic (e.g. continent, country, data center, etc.) →
As far as I gather, this subnet is already in violation of the predefined maximum number of nodes within the same country, which is supposed to be 2 for a 13 node subnet.
This proposal brings this number closer to the acceptable limit (though I believe the outcome will still be in violation of the acceptable limit, 3 instead of 2).
The above subnet information is illustrated below, followed by a node reference table:
Map Description
Red marker represents a removed node (transparent center for overlap visibility)
Green marker represents an added node
Blue marker represents an unchanged node
Highlighted patches represent the country the above nodes sit within (red if the country is removed, green if added, otherwise grey)
Light grey markers with yellow borders are examples of unassigned nodes that would be viable candidates for joining the subnet according to formal decentralisation coefficients (so this proposal can be viewed in the context of alternative solutions that are not being used)
I’m planning to adopt this proposal as the improvements to subnet decentralisation (in terms of jurisdiction diversity) are more important than the slight increase in average node proximity. I do have some questions though. @SvenF would you be able to answer any of these?
Am I correct in observing that the subnet is currently in violation of the predefined decentralisation coefficient relating to the maximum number of nodes that should belong to the same country? (and that the subnet will still be in violation of this even once this proposal passes, though it will be in a better position than it is now)
There are plenty of viable nodes located in South Africa and Australia (illustrated on the map). Any of these would have improved the country coefficient, as well as increasing the average geographic distance between subnet nodes, as well as increasing the number of continents that the subnet is composed of. Am I missing something, or would one of these have potentially been a better choice of node to add to the subnet?
When selecting nodes to add to a subnet to optimise decentralisation, is this done in a greedy fashion (considering just the subnet in question), or are redundancy requirements for other subnets also taken into account?
As a side note, there are two nodes (in Switzerland) in this subnet that are very close to one another, roughly an hours drive apart (but of course these belong to different node providers and data centre owners, otherwise this would also be a violation of the decentralisation coefficients).
@sat shouldn’t these coefficients be influenced by this proposal (as described in my post above), or have I misunderstood something? Are you able to share some pseudo code for the formula that generates these coefficients? I’m a little confused by the outcome of the country one.
The algorithm actually focuses on the Nakamoto coefficients across different features/dimensions. Here is what ChatGPT just generated when given the following prompt:
Can you please write pseudo code for calculating Nakamoto coefficients across 5 features/dimensions: "NodeProvider", "DataCenter", "DataCenterOwner", "City", "Country", "Continent"? This is for an Internet Computer subnet, and subnets on the IC have 3*f + 1 nodes, where f is the failure domain.
Generated code:
function calculate_nakamoto_coefficients(nodes):
# Dimensions to consider
dimensions = ["NodeProvider", "DataCenter", "DataCenterOwner", "City", "Country", "Continent"]
# Dictionary to store Nakamoto coefficient for each dimension
nakamoto_coefficients = {}
for dimension in dimensions:
# Step 1: Group nodes by the current dimension
groups = group_nodes_by_dimension(nodes, dimension)
# Step 2: Calculate the number of nodes in each group
group_sizes = []
for group in groups:
group_sizes.append(count_nodes_in_group(group))
# Step 3: Sort groups by size in descending order
group_sizes.sort(descending=True)
# Step 4: Determine the minimum number of groups needed to control more than f nodes
cumulative_nodes = 0
f = (total_nodes(nodes) - 1) // 3
for i in range(len(group_sizes)):
cumulative_nodes += group_sizes[i]
if cumulative_nodes > f:
nakamoto_coefficients[dimension] = i + 1
break
return nakamoto_coefficients
function group_nodes_by_dimension(nodes, dimension):
# Create a dictionary to group nodes by the specified dimension
groups = {}
for node in nodes:
key = node[dimension]
if key not in groups:
groups[key] = []
groups[key].append(node)
return groups.values()
function count_nodes_in_group(group):
# Return the number of nodes in the group
return len(group)
function total_nodes(nodes):
# Return the total number of nodes
return len(nodes)
# Main function to calculate Nakamoto Coefficients across multiple dimensions
function main():
nakamoto_coefficients = calculate_nakamoto_coefficient(nodes)
# Return or print the Nakamoto coefficients
return nakamoto_coefficients
# Example structure of a node in the nodes list
# node = {
# "Continent": "Europe",
# "Country": "Germany",
# "City": "Berlin",
# "Data Center": "DC1",
# "Node Provider": "ProviderA",
# "power": 100 # Power could be hash rate, stake, voting power, etc.
# }
# Example usage
nodes = [...] # List of node dictionaries with the above structure
nakamoto_coefficients = main()
print(nakamoto_coefficients)
The actual code we use is at:
And I also asked ChatGPT to compare the code it generated with the one in our code base, and check for bugs etc, and it answered the following:
Based on that, I’d say the implementation is (relatively) okay.
Am I correct in observing that the subnet is currently in violation of the predefined decentralisation coefficient relating to the maximum number of nodes that should belong to the same country? (and that the subnet will still be in violation of this even once this proposal passes, though it will be in a better position than it is now)
@Lorimer The Nakamoto coefficient is calculated in a slightly different way. You need to sort countries based on the number of nodes in that country, and then keep summing up the countries until you get over 4 nodes. So in this case the calculation in the code is correct. The NC is 2 for the country, both before and after the replacement. You seem to have an off-by-one error in your analysis. But we are improving the node distribution with this proposal.
There are plenty of viable nodes located in South Africa and Australia (illustrated on the map). Any of these would have improved the country coefficient, as well as increasing the average geographic distance between subnet nodes, as well as increasing the number of continents that the subnet is composed of. Am I missing something, or would one of these have potentially been a better choice of node to add to the subnet?
These nodes would likely make some other coefficients / dimensions worse.
When selecting nodes to add to a subnet to optimise decentralisation, is this done in a greedy fashion (considering just the subnet in question), or are redundancy requirements for other subnets also taken into account?
We are doing in greedy fashion, but only 1 node at a time. What does that mean? Let’s say that we need to add 2 nodes to a subnet. For each of the 2 nodes, we do:
calculate the Nakamoto coefficients we would get for each of the ~500 nodes
sort the candidates based on their Nakamoto
pick the best Nakamoto
find all candidates that have the best Nakamoto (may be 10-20 nodes with the same result)
pick one of the best candidates - since this PR we try to keep the same nodes if there is no improvement
Thanks @sat, my last comment was a bit of a brain fart (and certainly was caused by an off-by-one error, in my head). I expected the country coefficient update to look like this,
e[2m country: 1.00 → 2.00 (+0%) e[0m, rather than this,
e[2m country: 2.00 → 2.00 (+0%) e[0m,
because I was considering 4 malicious nodes to be needed to take down a subnet (such as the 4 in the US prior to this proposal executing), whereas 4 is of course the max number that can be tolerated (not the min number that cannot be tolerated).
That being said, I stand by the points that I raised in my main post. In order to comply with formal subnet limits, the Nakamato coefficient for the country characteristic needs to be 3. Hence 2 is too small (requiring the nodes of only 2 countries to collude in order to theoretically attack the subnet). This is documented in more detail here →
^ Given this, I would reiterate my point.
I believe an additional proposal is needed to get this subnet back within the acceptable limit according to the current and revised target IC topology (a limit of 2 nodes per country, meaning a minimum country Nakamoto coefficient of 3).
I wouldn’t be surprised if other subnets have similar issues (I’ll run some analysis when I get a chance, maybe tomorrow).
Regarding the unassigned South Africa and Australia nodes, I was already filtering out nodes using the strictest of constraints (only returning unassigned nodes that did not share a single characteristic with the existing subnet nodes i.e. continent, country, node provider, data centre, owner).
These nodes could therefore only have had no effect or a positive impact on the Nakamoto coefficients of the subnet (not a negative impact). Here are a handful of example candidates:
This approach is problematic. For example, none of the unassigned nodes illustrated on the map that I rendered above would result in an improved Nakamoto coefficient by themself (for anything other than continent). This doesn’t mean that they wouldn’t improve decentralisation (essentially making it easier to improve the other Nakamoto coefficients with subsequent node swaps in the future).
I think the crux of the issue is that the Nakamoto coefficients are throwing away information because they’re discretised. This is demonstrated by the fact that they weren’t even affected by this proposal. I think it would probably make more sense to optimise for subnet limits directly, and therefore indirectly optimise for the Nakamoto coefficients (so that all information is available and considered during the optimisation procedure).
An empirical observation: we very rarely have enough nodes decentralized nodes to have a Nakamoto of 3 for country.
We show the Nakamoto at the end, but we sort the candidates based on other values. So your suggestion (if I understood it correctly) is already done. If not, please send a PR
Am I missing something? I also count 31 nodes that don’t share any of these characteristics with this subnet (I shared 4 of these in my previous message).
If I were to raise a ChangeSubnetMembership proposal myself to resolve this nonconformance, assuming that it achieves the desired Nakamoto coefficient across all dimensions for this subnet, would this be a proposal that DFINITY would be happy to accept?
Hi @Lorimer I think your observation is correct that there should be enough nodes to Nakamoto coefficient of 3 per country (since country limit is 2 for most subnets, which would mean 5 nodes need to collude to take over a 13 node subnet, which would be 2 in same country + another 2 in same country + any other node, which means Nakamoto coefficient is 3). Otherwise the IC target topology would not have been reached.
But to support the statement of @sat, country is the most limiting factor in decentralization (there is much more variety in Node Providers, cities, data centers, and data center operators than countries), and spare nodes are not taken into account in the optimization tool, so suboptimization by seeking the maximum decentralization for one subnet needs to be avoided.
In the example above, swapping two nodes (for example adding one node from a Node Provider in South Africa, next to replacing the unhealthy node) is indeed interesting but does not change the nakamoto coefficient, as Sasa also indicated. With 3 (instead of 4) US nodes, and 2 nodes in Switzerland you still only need to have control over these two countries in order to take over the subnet. What is more interesting is the add another - third - node that replaces one American or Swiss node, then the country limit would bump of from 2 to 3. The current dre-tool can be run multiple times so that you can continuously improve each subnet, but for the two reasons above (country limit, and need for enough spare nodes) it’s better to only do this increased optimization for the larger (i.e. high priority) subnets such as NNS, SNS, II and Fiduciary subnets.
Please have a look at the dre-tooling and happy to hear your throughts.
Do I understand correctly that you’re saying there’s an intentional plan to allow the smaller subnets to deviate from the formally proposed and voted in target IC topology, in order to give the larger subnets a better chance of meeting those same targets?
Doesn’t this imply that the picture that’s been painted for the current IC Target Topology proposal is inaccurate and misleading, as this will lead to a scenario where the smaller subnets are even less likely to be able to meet their formally voted in topology target? (unless more node providers in more countries are onboarded)
I get the impression that the tool has issues when used in this way (given the payload problems that have occurred with these four recently submitted ‘Change Subnet Membership’ proposals - 5kdm2, mpubz, tdb26, uzr34).
I’m quite interested in writing an alternative tool from the ground up using a different optimisation approach (non-greedy, non-linear, stochastically optimised using something like a simulated annealing algorithm - or whatever does a better job of avoiding local minima). I’m also interested in optimising for the fewest number of swaps required to meet arbitrary targets.
Does this sound like something that would be useful (I’d open source it of course)?
Yes, absolutely. With this in mind, last week I’ve created a what-if subcommand in the DRE tool, which should allow you to easily play around and see what kind of an effect on decentralization would some replacements make.
My suggestion would be to make sure you run this before submitting a proposal. Otherwise, you might get a rejection for something that seemed like a good action.
DFINITY will submit a proposal in three days on Monday to reduce the notarization delay on subnet 4zbus, similar to what has happened on other subnets in recent weeks (you can find all details in this forum thread).
).
)
)
(but of course these belong to different node providers and data centre owners, otherwise this would also be a violation of the decentralisation coefficients).