The State and Direction of Decentralization & Nodes on the Internet Computer

Roadmap
128

Update November - Node Decentralization

Following up on the roadmap that we posted four weeks ago, I would like to give an update of the decentralization activities since then. Before going into the details, let’s look at the network topology as of today versus last year. A lot of progress has been made with respect to decentralization so far. The table below shows that overall decentralization has increased over the last three quarters on every metric, with the exception of the number of Node Providers (NPs) and Geographies, which is the reason our target for the next year will be to have new Node Providers onboard, in particular in new Geographies.

1600×725 116 KB

Storage increase of existing replica nodes

To recap, Node Providers are currently upgrading their existing configuration, allowing these Node Providers to independently manage their nodes, while at the same time increasing the storage capacity of the nodes from 3.2 TB to 32 TB. This upgrade is in full progress, with Asia nodes being 100% completed, European nodes 94 % (1 Node Provider left) and US nodes 44% completed (9 Node Providers left). We expect the upgrade of existing Node Providers to be fully completed by the end of the year.

When the upgrade of existing Node Providers is completed, the Dfinity team that is consulting these Node Providers can start to focus on the autonomous onboarding process for new Node Providers.

Testing of 2nd Generation Hardware

The minimum requirements for the next generation of replica node hardware (Gen2HW) have been published on the Internet Computer wiki, as well as three specific configurations of HW that have been validated by the Dfinity team. Validation of other HW specifications is in progress, and for discussion please refer to the separate Forum post New Hardware Specification.

Testing of the fully autonomous NP onboarding process

The new NP onboarding process consists of two parts, covered by two separate runbooks: the onboarding of a new Data Center (DC) and the registration and acceptance as a new Node Provider by the NNS. The latter runbook was already available on the wiki. The DC runbook is completed as well, and is currently being verified internally. The next step will be to test both the NP registration runbook and the DC setup in combination with Gen2HW. A brave potential new NP has agreed to test this process in November and December. The progress of these tests will mainly be determined by the lead time for procuring specific Gen2HW.

Remuneration

Cycle pricing and node provider rewards (remuneration) require refinement as the network grows. More concretely, we propose the following changes with respect to node provider rewards:

  • Today’s Genesis remuneration model: In early 2021, NPs had to commit to a not-yet-launched network. To compensate for the risk taken, the original genesis remuneration model includes significant profit margins. This model does not cover all of the storage extensions described above, so the foundation has been covering a portion of those costs, while the storage was not benefitting the network.

  • Interim Gen2HW-ready remuneration model: While a more general remuneration overhaul is required and needs to be discussed mid-term (see next bullet point), an interim update is required as the storage upgrade comes to a completion and NPs using Gen2HW are onboarded. We propose the remuneration to be take into account two factors. First, HW costs have risen and need to be compensated. However, having a running IC network reduces the risk of entry for new NPs, hence a lower risk premium is being included in the reward. Taking into account these above factors, the interim remuneration is lower than the Genesys remuneration model. The wiki page (Node Provider Remuneration) shows the draft of the interim remuneration model that we propose to use. It is based on currently available hardware prices and average rates for ICP and XDR. As shown from the table, the premium multiplier for remuneration will lower from 2.5 to 1.9 to take into account the reduced risk premium for joining the IC network. We look forward to your feedback in this forum thread, plan to incorporate it into the interim remuneration model and then submit the respective proposal to the NNS.

  • Future remuneration model: We acknowledge that the interim remuneration update does not address all open questions such as how to incentivize decentralization through an adaptive remuneration. A more general update needs careful preparation and a thorough discussion with the community. We plan to start that discussion during the next few weeks.

Node provider self-declaration

As already asked by Jordan Last about a year ago, the autonomous NP onboarding process requires incentives for NPs and node operators to act in the interest of the IC. Moreover, the community needs sufficient information in order to be able to decide whether to adopt or reject a new NP. As sketched by Dom on Medium, we plan to discuss and propose a self-declaration for NPs. We will present that in a separate thread shortly. Stay tuned.

Way forward for onboarding

With the runbooks tested and with a proposal for interim remuneration, we can start onboarding new node providers in a controlled way. Currently, the onboarding process requires sound technical knowledge, which is something we are aiming to reduce in the coming weeks and months. The steps to follow are roughly as follows:

  • NP to prepare self-declaration (separate thread to follow shortly)
  • NNS registration through proposals (see Wiki instructions including all prerequisites)
    • Register as an NP (including pointer to self-registration)
    • Create a new data center record (if that does not exist already)
    • Create a node operator record (if that does not exist already)
  • NP orders hardware (see verified 2nd gen replica node configurations on the wiki)
  • NP/node operator setup DC, install nodes, set up the reward configuration and have them join the network.

Lead time for ordering hardware can be up to several weeks or even months, thus the above process will take time. Most of the work is with the node providers: preparing documents for self-declaration, submitting proposals, purchasing hardware, setting up the nodes, etc. As outlined in earlier posts, we plan to provide a better user experience for some of these activities, especially for submitting proposals and changing the NP profile. For now, however, NPs should be comfortably working with a console.

As explained in the introduction of this post and also motivated in the Medium article Let’s Grow the Internet Computer Network, the IC network needs to grow. In particular, the network needs

  • more, i.e. new, node providers and
  • nodes in underrepresented geographies, such as Asia, Australia, and Africa in order to improve decentralization.

Call for action

In summary, we encourage

  • new node providers
  • who are technically savvy and not afraid to be among the first to run the autonomous onboarding process
  • and plan to contribute nodes in underrepresented geographies

to start reviewing the hardware specifications, to familiarize themselves with the runbooks, and to stay tuned for the upcoming self-declaration discussion.

We propose to start with small batches of node machines (e.g. 5 per new node provider and location). As the process matures based on the feedback of these early adopters, we expect the numbers to grow more rapidly.

We look forward to feedback from the community, especially from potential future node providers.

30 Likes
129

QQ: What defenses does the network have against bad node providers?

What if we onboard a bad node provider who only violates the protocol? How will people on the NNS know if a node is bad? Are node rewards slashed?

8 Likes
130

From what I’ve gathered there is an SLA as a deterrent for downtime.

As far as validating node specs beyond simply being hard down, not completely sure.

2 Likes
131

Thank you for sharing the plan.

A few questions:

  1. The WIP Gen2 renumeration model assumes an annual inflation rate of 10% when estimating operating expenses over the next 4 years. That seems overly pessimistic. According to the IMF, global inflation will be 8.8% in 2022 but decline to 6.5% in 2023 and to 4.1% in 2024. Also, developed countries typically experience less inflation than developing countries; I wonder if a more fine-grained inflation model makes sense here.

  2. On the topic of OpEx, I am curious why Asia OpEx is nearly twice USA OpEx. The wiki says those estimates are based on numbers shared by current NPs. Wouldn’t NPs have an incentive to inflate those numbers to receive larger rewards? What checks and balances do we have to prevent NPs from lying? Could the NNS leverage the new HTTPS outcall feature to query external sources without relying solely on NP claims?

  3. It is difficult for a layman like myself to assess whether a 90% profit margin is appropriate or not. How does this compare with other blockchains, e.g. validators on Ethereum? How does this compare with cloud providers? Should the margin be automatically adjusted based on some formula in the future?

  4. Somewhat unrelated, but this wiki page says “Renumeration of node operators” instead of “Renumeration of node providers”. I’m assuming that is a typo.

Thank you!

10 Likes
132

I’m bumping this because this is an important question. How can we ever add non-Dfinity-approved nodes if we don’t have a guarantee of protocol disincentives?

2 Likes
133

Add badlands to the roadmap!!!

2 Likes
134

@Mr_Burkes @plasmex these are very important questions indeed. we can automatically check some elements in advance (see also this other forum post: Proposal for Node Provider Self-declaration) but there is of course still to potential for violating the protocol. There are two features we are discussing and preparing internally, work in progress: an automated reward system in the NNS that includes penalty for policy violation, and observability/monitoring of NP performance. Both are on the roadmap and being worked on.

2 Likes
135

Hi @jzxchiang thanks for the detailed feedback. To respond to each of your items

  1. correct, we have taken a conservative estimate, taking into account that we want to stimulate NPs in new geographies to join.

  2. Yes there would be incentive to inflate the numbers but we have a good view on the operating expenses based on the existing NPs and the operation of the Dfinity nodes. And yes, node provider performance can to some extend be monitored automatically as well; this is a feature we are planning to implement in the future as well.

  3. it is very difficult to compare with other blockchains as the business model for validators and miners is completely different. For the IC, a NP does not get rewards for number of transactions run on a node or subnet, but for performance/availability of the node. The intention indeed is to have the reward system automated inside the NNS and be updated automatically. To be continued and posted in the forum once we have suggestions for this.

  4. Yes will correct this.

4 Likes
137

To add, the above are future features we are planning to add. Note that in the current implementation, we need consensus of at least 2/3 of the nodes in a subnet to continue. A node that is deviating is not removed automatically and collects rewards, but it should be eventually removed.

3 Likes
138

@Mr_Burkes I want to escalate this point by @SvenF because it directly addresses your question about misbehaving nodes, and I am afraid it may be buried among the comments.

3 Likes
139

Thanks for the response. Regarding (1), I wonder if the profit margin should be enough to incentivize NPs. I’m not sure the inflation rate is a parameter that should be tuned for anything other than accuracy based on official projections.

2 Likes
140

What does eventually removed mean?

2 Likes
142

@Mr_Burkes , the nodes on every subnet are monitored, and the status of each node is visible in the public dashboard (e.g. NNS subnet status). If a node is down or unhealthy, it is eventually swapped for another, healthy node. This process is done through an NNS proposal, as you can also find in the dashboard under governance (see here an example of a proposal for a node replacement: Proposal: 93959) that the community votes upon and is eventually executed.

5 Likes
143

@SvenF I think the rewards overall are too generous and reward rate for Gen 2 are too close to Gen 1. ICP should have plenty of willing NP’s to come in at a rate under ~1.9x (proposed Gen 2 USA renumeration) regardless of geography. I don’t think there will be significant or any drop off from Gen-2 NP interest if we reduce to proposed renumeration to something more reasonable, perhaps ~1.5x.

3 Likes
144

I don’t know why the process to on-board a new node must be so convoluted and difficult. This seems to me to be in contradiction to the claim that “The IC is designed for anyone to be able to become a node provider (NP).”

I am an individual with the resources to invest immediately in one or a few nodes to start with, and in at least two geographical regions (datacentres in cities not yet served/in the list) and I would have hoped to expand the operation in the future, but the process seems impossible. For example, elsewhere it is recommended that the proposal contains details of my github (or similar) page and my business website. Well:

  • I do not have a github page because I am not a programmer (but I do have sufficient experience to run a node);
  • I have not formed a legal entity to own and run the nodes, and therefore do not have a business registration or business website. I would want to do this (spend time and money on creating the business structure) after having my proposal approved.
    So I am supposed to stake x ICP for the slim chance of the proposal being approved in the circumstances described? Obviously it (the minimum stake which appears to be ICP 10.00) is not a lot of money right now, but it seems almost certain that I would be simply throwing that money away.

I wanted to approach this in good faith and be rewarded for my non-statistically-deviating node(s), but the barriers are too high. I suppose I have to look at opportunities elsewhere, which is a shame, as I can see that IC has amazing potential and it is disappointing that it seems I cannot contribute to its future in one of the few ways I can. The rules/processes, it seems to me, aren’t doing much for IC’s resilience/decentralisation. Disappointing!

6 Likes
145

@SvenF would you consider simplifying the rules to help out this fine gentleman / lady? Onboarding new NPs should be straightforward.

2 Likes
146

Hi @SS88 and @dfisher, thanks for the feedback and agree that the onboarding process and node setup needs to be straightforward!

The node setup is still quite tech-savy and it will take some time in order for the setup to be done without any technical knowledge, but that is definitely something for the long term roadmap.

But the onboarding process can of course be further simplified to include non business entities. That’s why the proposal for the onboarding process was posted on the Forum. Eventually, it should of course be for the community to decide what would be the onboarding rules for a new NP. For a person that is not a business entity, the community should have some way to identify the authenticity of the person. Since that person does not have a github account or chamber of commerce number, some other document or way of identification might be required. Any suggestions is very much welcome and I am happy to pick this up together with you.

4 Likes
147

It is better to be safe than sorry!

148

SvenF and dfisher, thanks for your replies.

I am an IC beginner, but I have read the articles about nodes meeting the correct specification and so on and I don’t really understand why it is a concern; after all, if the node is not meeting the required distribution then why not just slash it? I accept that I might be missing some technical or theoretical understanding here and don’t spend time replying as this is more rhetorical but I have talked to others who have similar questions.

Also, I have read in other places comments like:
“You need Dfinity’s permission to run a node!”
“It’s not decentralised, Dfinity runs the network!”
“It’s highly centralised, there are only 8 node providers!”
“It’s a scam because [some variation of the above, or comment about early investors]!”
etc.
and I think that this difficulty of adding a node really does contribute to this bad press. Therefore I do hope that a great reassessment and simplification of the process can be made because, now, people like me who have good intentions are shut out.

At the moment, the barriers are too high for me (and I understand that sufficient nodes are lined up going into 2023 anyway), so I am going to sit this out until the process is a bit easier.

Thanks again for the quick replies.

4 Likes
149

Thanks @SS88, completely agree with your observations getting a node up should be much simpler and the team is working hard on this, as well as to get it further decentralised. As you can see from previous updates there has been progress to decentralise the node infrastructure, and this will be focus for 2023 as well. If you are still up for it now, I am happy to help you out of course. In the meantime I will continue to post updates on how we are doing with the decentralisation.

2 Likes