Relevant update: Improvements to Node Provider Remuneration
4 Likes
Out of curiosity, is there any protocol that would stop a single country having enough nodes (albeit with independent parties running them) to disrupt the IC? Right now according to the https://dashboard.internetcomputer.org, the over 50% of nodes are in the U.S. (see totals below in post).
Imagine what could have happened if the IC had launched back in 2018 and a ton of nodes spun up in China, then the government cracked down.
I’d love it if we could flush out a few ideas on this or start up a proposal to monitor node concentration by country and make sure it doesn’t ever reach a point where a single country could take the network offline.
Here’s my quick count from https://dashboard.internetcomputer.org on the current status of nodes per country:
United States: 210 nodes total
• Portland - 17
• Chicago - 16
• New York - 11
• Atlanta - 23
• Atlanta2 - 13
• Jacksonville - 17
• Orlando - 22
• Tampa 20
• Dallas - 21
• Las Vegas - 11
• Fremont - 19
• San Jose - 20
Belgium: 48 nodes total
• Antwerp - 19
• Brussels - 19
• Brussels2 - 10
Germany: 27 nodes total
• Munich - 27
Switzerland: 58 nodes total
• Geneva2 - 21
• La Chaux-de-Fonds - 20
• Zurich2 - 13
• Zurich3 - 2
• Zurich5 - 2
Romania: 25 nodes total
• Bucharest - 25
Singapore: 47 nodes total
• Singapore - 19
• Singapore2 - 9
• Singapore3 - 19
5 Likes
Hi @icme.
Thanks for bringing this up.
The reason why we currently have a slight majority of nodes in the US has historical reasons. The hardware was provided directly by US manufacturers (Dell and Supermicro) and the US based node providers got them delivered first. This all happened with the third covid wave. International shipments and the coordination of remote hands was very complicated. We rerouted hardware for our own test networks that was already in Europe and Asia but this wasn’t sufficient to balance out the US number of nodes.
In the meanwhile the hardware has arrived in all data centers worldwide. Since autumn 2021 were are only onboarding European data centers. We had three in January (ZH3/5/7) and four more are following in the next two weeks (Marseille, Ljubljana, Maribor and Frankfurt). Two data centers in Tokyo are expected for March to extend the number of Asian nodes.
The remaining data centers to the full node count of about 1300 nodes are mixed. On the end we will still have a minority in Asia and big black spots in the rest of the world which will be closed by the new, decentralized node provider onboarding mentioned in this post.
In the meanwhile we are doing two things, first we are replacing US nodes from extings subnets with nodes from new European node providers, the first three subnets we did last week and more will follow every week, and secondly we are preparing a decentralisation API for the public dashboard. This API is based on the one that our SRE team is using since launch to decide about the best possible decentralisation of new subnets using the Nakamoto coefficient. It takes the node provider, region and gps coordinates, and data center owner/carrier into account. The public dashboard will show the decentralisation index for each subnet and makes the improvement visible to the community. Of course we will publish the corresponding code to allow ambitious community members to check the decentralisation in proposals for new subnets. The improvement of the decentralisation will not only be provided by more diversity but also by an expected increasing number of nodes per subnet.
I think that the IC proves real decentralisation much better and earlier than other projects of the same size. But I also think that this “decentralisation quality” will be hard to keep. The main reason is the lack of small data centers and the increasing number of data centers provided by a defacto oligopol. But we are aware of that and see it as a challenge when establishing a decentralized infrastructure that builds on professional datacenters. One way to get more diversity in the available data centers owners could be to pay higher rewards for nodes from small datacenter that very often also have higher costs.
18 Likes
This is extremely exciting. Also this whole post beyond the quote really shows the care DFINITY has taken thus far to ensure decentralization in the network. This is excellent.
I am also worried about maintaining the quality of decentralization, especially as data centers and nodes can permissionlessly onboard. For example, how do we verify location and independence? For gps coordinates, are there TEE GPS hardware modules that can sign coordinates?
If we can’t verify the essential information necessary for decentralization, then the network could begin to succumb more easily to Byzantines.
10 Likes
Thanks for your feedback. I’ll forward that to the team. We really appreciate such feedback and are looking forward discussing and not only reporting the challenges that we are seeing.
I am also worried about maintaining the quality of decentralization, especially as data centers and nodes can permissionlessly onboard. For example, how do we verify location and independence? For gps coordinates, are there TEE GPS hardware modules that can sign coordinates?
There are many things that could endanger the quality of decentralisation. As I explained before the only real decentralisation is the decentralisation that can be verified/measured deterministically. But this means that we need to trust the data that is used. With the first generation of node providers this was easy as we could literally trace the shipments to the datacenters and had personal contact to the node providers. With the next generation of node providers this will change. It’s not completely permissionless, a node provider (NP) still needs a proposal to get registered and needs a registered datacenter (DC) to link a node operator (NO) to onboard nodes. This means that we are dropping the need of a proposal for a new node operator if the community is able to verify the NP and corresponding DC.
The location that is proposed is relatively easy to verify and won’t change for a DC. But there are only a few ways to verify that the nodes are actually running in the DC that was linked to the NO. GeoIP lookups and hop counting are the two that we are implementing atm. Ideally the DCs would register themselves and ensure that there no “malicious” links from NO to their DC record.
The independence of a node provider is hard to check and it could obviously change by time. Proposals for new NP will be able to be created via the NNS Frontend Dapp and will have a summary field where NPs are asked to explain their motivation and qualification. We’re currently discussing which guidelines we want to give to the community to decide about new node provider’s proposals. Nevertheless we can’t lonely rely on an initial assessment of the indepence of a new node provider. To address this issue we’re reducing the number of nodes that a new NP can onboard initially which leads to a higher diversity of NPs per subnet. Another idea that is discussed is to shuffle nodes in subnets randomly. Even if there would be a cohort of NPs it would never exist for a longer period within a subnet and they won’t be able to organize an attack against the integrity of subnet because they can’t predict when the cohort will have a majority in a subnet.
A GPS hardware won’t always help as they aren’t working in every location. In addition it would be a huge effort to run them trustworthy and we would need to provide different models to ensure the decentralisation in the supply chain. This problem we already have with our Nitrokey HSMs, the only open source HSM that we found on the market, and that’s why we are introducing an alternative way to register NO without using an HSM device. Special HW makes decentralisation always more complicated.
If we can’t verify the essential information necessary for decentralization, then the network could begin to succumb more easily to Byzantines.
The DC details (GPS coordinates and legal person) are the essential information that will need to be verified by the community. This covers 3 of 4 decentralisation parameters. The risk by malicious NPs can be mitigated in many ways. I wouldn’t be surprised if the need for NP proposals would be dropped when all controls are in place that reduce the rest risk of malicious NPs to an acceptable level.
6 Likes
I am extremely impressed with the level of thought you are taking to this, and I feel we are being very well cared for.
Will continue to engage along with the community as we strive to ensure decentralization and thus the most secure network possible.
6 Likes
I’m hoping all are starting to see how powerful node shuffling can be. Are there dissenting opinions on this still? What are the current arguments against this?
6 Likes
I’m not sure how widely this was discussed in other teams. Since beginning of the this year an orchestrator team was splitted out of the consensus team. I guess this team will take the lead on such a feature.
Node shuffling are topology changes that currently are done manually via proposals by the SRE team. For decentralisation improvements that’s fine but for random shuffling this would need to be NNS driven. I guess that other topology management related features like subnet splitting could get prioritized higher. The only reason that I could imagine why the random shuffling wouldn’t be done is that we found something that is even better, less load intensive. Shuffling nodes in subnets means syncing the state of a subnet which can already cause hundreds of GBs of traffic.
2 Likes
I am extremely impressed with the level of thought you are taking to this, and I feel we are being very well cared for.
This sentiment probably summarizes why many 8+ year stakers haven chosen to stake for 8+ years.
I’ve literally seen nothing about the IC that was carelessly designed, and I’ve read hundreds of posts on this forum. There are known knowns and known unknowns—but almost no unknown unknowns (that I can see).
12 Likes
Hi, I am beginner on this forum, this is my first post.
How to become a node provider in practice? I must lease server in cloud or opposite, I must buy computer for my own and use it in data center. What is cost? How much ICP constant in fiat I can get for this node. If after years server will removed from neural net, I can use it for my own as my private computer or sold it?
Hi Koto and welcome to the node provider community.
How to become a node provider in practice?
You need to wait until the NNS frontend provides a way to register as a node provider. This is expected for end of Q2 2022. Once your proposal is accepted by the community you can create a proposal for a new data center record or choose one that is already in the registry. Once both, the node provider and data center record, are in the registry you can set up and deploy your nodes.
I must lease server in cloud or opposite, I must buy computer for my own and use it in data center. What is cost?
IC nodes are running on professional data center infrastructure. Cloud instances can’t be used for running an IC node. Also by end of Q2 2022 we want to present the new node hardware together with some hardware manufacturers. The costs aren’t clear yet. The last node types were about 10K$. I personally expect the new nodes being in a similar price class as we aren’t reducing the specification. But there is a risk that current issues with global supply chains could even increase the cost per node.
How much ICP constant in fiat I can get for this node. If after years server will removed from neural net, I can use it for my own as my private computer or sold it?
Details about the current node remuneration you can find in this forum post.
The nodes that you onboarded will get rewards once they were added to a subnet. The life cycle of a certain node type isn’t meant to be limited. The IC allows single nodes to die without any impact on the services running in that subnet. As long as the IC has enough nodes of a certain node type to create a subnet these node types will be used and get rewards. The sustainable usage of resources is important to the IC and differentiates again the IC from classic cloud computing. That said the efficiency of server hardware is constantly improving and rewards will need to decrease the more new, more performant types are introduced. Failing hardware will also cause penalties which again can reduce the rewards. This together could lead to node providers deciding to replace their hardware and do whatever they want with the old hardware.
5 Likes
Hi Luis. Where can I registry a new datacenter here in Valencia-Spain? While Q2 arrive with new hardware requirements… Thanks!
1 Like
Valencia 
my home town. I would love to see nodes there but the IC unfortunately doesn’t care about individuals like me 
We have data centers in Barcelona and Madrid from the first generation node providers waiting in the backlog. They’ll come in the next few months 
New data centers will be able to be proposed once the new onboarding workflow is provided by the NNS frontend like I explained in the previous post.
4 Likes
Che Luis!! Fallas is here these days!! Jajajajaja!! Ok, where can we follow updates about the dates and ways to make it?? Here in the forum?? Thanks!! Gracies!!
1 Like
I have to skip Las Fallas this year but looking forward to a good paella de marisco in May
Yes, we will update this thread with every milestone that was reached.
2 Likes
I’m in! Thanks Luis!!
1 Like
@diegop Hello, I would like to ask, if I want to be a difinity network node, where can I get the deployment program of the node
2 Likes