The State and Direction of Decentralization & Nodes on the Internet Computer

@diegop Have time to answer or not? You may just ignore the question? lol…

Can you share this kind of roadmap with the #ICP community? or maybe just provide an updated version here?

What do you guys think about putting some of the node machines on satellites? That way we can even decentralize the telecommunication services. In the future, when telecommunication is further merged with computing, IC would be much more powerful than our current imagination.

Check SpaceChain - Decentralized Space Agency

Dear Node Provider Community.

I’m happy to update you about our progress on the M1 milestone of the IC platform decentralization roadmap.

We published a wiki article describing the current decentralized onboarding process for node providers and asked an existing node provider in the US to onboard himself once again to verify that this process is working for external parties too. As we explained before, the first generation of node providers was onboarded by us. You will see the corresponding proposals in the NNS in the next few days. Just be aware that this isn’t actually a new node provider and therefore all changes will be reverted once the verification is done.
We also implemented a CI job that checks that this process is working with every change on depending components and that’s almost every component of the IC :slight_smile:
The instructions show a lot of potential for improvements and there will be a lot of changes to that process in the next months. We are already working on adapting the process to the new node remuneration process that only considers contributing nodes instead of the full node allowance of a node operator.
We are also working on implementing the onboarding process to the NNS Frontend Dapp to provide a UI supported onboarding process for node providers. This will include some simplifications like creating node operators without the need of a proposal.
The first parts are planned to be rolled out by latest end of June. The full feature coverage is currently planned for end of Q322.

Another huge effort is the decentralisation of the datacenter setup. The same node provider that is running the verification that I previously explained will also doing the first acceptance test for the new data center setup instructions. The requirements listed in the instructions above already give a hint how trivial our datacenter setup is planned to be in the future. We really minimized to the max. All the equipment that we needed since launch to gather metrics in order to improve the protocol isn’t needed anymore. The acceptance test that this node provider will run is to verify that the instructions are allowing the removal of the equipment in order to reduce the setup to what’s needed in the future.
In the next days you will see many proposals in the NNS replacing and removing of all nodes in TP1 (Tampa) in preparation for this acceptance tests.

What’s next?

Assuming that all acceptance tests passed we are going to notify all existing node provider that haven’t onboarded their nodes yet (~800 nodes) and ask them to follow the corresponding documentation. Depending on the progress we will use new data centers to replace old data centers in order to get them redeployed in the same decentralized way. We are assuming that this will take the rest of the year and will end with ~1300 nodes running in the IC.
Somewhere in the next months we would like to announce the specification for the node type supporting AMD Milan to ensure that the network is growing with nodes supporting SEV-SNP attestation.

I was asked to build up a DeSRE team (Decentralized Site Reliability Engineering Team). This was never done before and I’m super excited that I got the chance to work on that. There will be a separate post about this but in short this means that we are planning to decentralize the way the IC is currently maintained. Not on the governance layer, that’s decentralized since launch, but from the operational side.

38 Likes

Umm…this is extremely exciting. Congratulations and thanks for all of the hard work on this.

15 Likes

This is really great news that go under the radar. I can’t wait to see what the IC is like with 1300 nodes.

6 Likes

Is the IC getting enough usage to support that many nodes? Is there any concern about the amount of rewards that would have to be paid to all those nodes if there is not enough cycles being burnt?

5 Likes

I can help here.

I am working with a research scientist on An analysis of this question and a post to show some concrete numbers, but the short version is that node provider rewards are a tiny percentage of rewards created (2% of rewards created total, NNS rewards are 98%). Therefore, we believe it makes sense to grow the infrastructure and node provider layer for now to grow in testing, resiliency and decentralization… but this is a short term thing, of course. It would not make sense to add 1 million nodes tomorrow for “resiliency” and pay that many nodes. We do believe it’s prudent to go to 1300 nodes and more, but ultimately of course the community will decide. If node providers rewards get too high, community could vote to not add more node providers.

I hope to have that post public next week.

8 Likes

Dear @808mafia,

This topic is on infrastructure layer of the IC. As the creator of this thread, I ask you please stay on topic. I hope my directness does not come off as rude, but I have to play the “direct guy” role in the forum to make sure the conversation stays at the technical level.

3 Likes

hey…relax it’s the weekend…take it easy…what ever fuck up ( or maybe not) you think you made is already done. Just take your Girlfriend or Boyfriend out for a drink, get a massage or something. stop watching CMC. you are not going to influence anybody in this forum. Smart people hang around here :slight_smile:

5 Likes

Marvellous!! Thanks for your hard work Luís and team! :handshake:

2 Likes

You guys are killing it. Decentralizing the onboarding process is the key to shutting down any concerns of node centralization.

Really proud of the work you guys are putting in. Can’t wait to spin up a node of my own one day :slight_smile:

5 Likes
  1. How much ICP does each node get paid on average?
  2. What is an expected cost to run an IC node?
  3. Are nodes in countries like US awarded more than nodes located in places like Indonesia? I ask this because operations costs can differ greatly depending on location i.e. exchange rates risks, utility costs, hardware etc

I’m asking because i’m interested in running some at facilities in my region (currently a region with no nodes)

  1. How much ICP does each node get paid on average?
  2. What is an expected cost to run an IC node?
  3. Are nodes in countries like US awarded more than nodes located in places like Indonesia? I ask this because operations costs can differ greatly depending on location i.e. exchange rates risks, utility costs, hardware etc

I’m asking because i’m interested in running some at facilities in my region (currently a region with no nodes)

When you remove yourself from this forum. So… do you REALLY want it to go up?

1 Like

Super exciting! (and kind of ‘under the radar’) . Great to follow your explanations, and thank you for the careful effort you put in to inform the community about node deployment/dev.
Cheers!

1 Like

I was asked to build up a DeSRE team (Decentralized Site Reliability Engineering Team). This was never done before and I’m super excited that I got the chance to work on that.

Very much looking forward to hearing your thoughts and plans on this front!


On a higher level, can you explain again the reasoning behind the distinction between data center operators and node operators? Isn’t the data center operator responsible for operating all the nodes in their data center?

Also, is it correct to say that most data center operators also register as node providers? I can’t imagine why someone would operate a data center without receiving ICP remuneration from the NNS (unless they directly charge the node providers who do).

Typically data center operators are a much larger concerns (10k-100k physical servers) that provide physical infrastructure to node providers…physical infrastructure such as land, building, physical security, power , back up power, some amount of isolation (cages in which indvidual node providers of different types of nodes can put their own servers).

Most data center providers (such as vantage for example) would NOT register as node providers. They rent the capabilities to node providers & the IC node providers in turn get ICP to pay the data center providers.

It is true that the data center operators need to provide certain guarantees to node providers (in terms of certifications) which the node providers use as basis to provide the nodes on top of the capabilities provided by data centers.

“Datacenter Operator” isn’t a entity that is relevant to the decentralisation of the IC. Sometimes they are also called “Remote Hands”. Both describes an entity that has physical access to the nodes. This can be the node provider itself or a company, mostly data center provider, the node provider contracted with.

Node provider are sometimes their own datacenter operators. Sometimes they pay the remote hands at the data center they contracted with.