The State and Direction of Decentralization & Nodes on the Internet Computer

[bjoern]

@ritvick

Is it still possible to become a NP and provide node to network and earn rewards? without steps A to D (mentioned in this thread ) being completed ?

It is (and will always be) possible to become a new NP, but there are a few practical obstacles at the moment. One is the hardware spec; as stated by @Luis above we are working on a new (updated and less restrictive) hardware spec; the old hardware spec has been very restrictive and it may not be straightforward to get new hardware satisfying it. (I’ll try to come back with more details on the new spec.) Another (potential) obstacle is that the onboarding process is quite technical at the moment and the tooling needs improvement.

The – technically – essential parts of the deliverables (reproducible builds, software distribution for deployment) have been completed. Practically, another dependency is then the new node hardware spec.

[esquivada]

If we want IC to be the L1 of the Internet, you must work hard on it and facilitate access, we need a good balance between descentr & safe of the chain

[ritvick]

Okay I see your point.
Now since we agree that any one following the wiki can spin up the node. I have few questions based on few assumptions
Assumption:

1. Lot of Votes are follower votes, so I understand if Dfinity foundation / ICA/ (whoever has majority of following ) votes then the proposal is expected.
2. Hosting a node requires few proposals, new NP proposal, New DC proposal, and few more.

Question:

1. If anyone submits the (“Register a New NP proposal” and other proposals) to add a node to network, on what grounds Dfinity foundation / ICA/ (whoever has majority of following ) approves or rejects it ?
2. I see a lot of nodes are sitting in Awaiting subnet status. And as per my understanding one does not get rewarded unless the node is in a subnet. What is the criteria to get into a subnet ( either old or a new one) ? . Are subnets created automatically or manually for now?

The whole point of asking these questions is though we wait for this thread to complete there are people who want to invest and become node provider, and it comes with a hefty cost.

Request: Whoever answers this post, please try to objectively answer the bullet points and all the questions if possible. I am sure me and community will have followup questions on this.

[jsull9]

Yes, personally I would like to see this happen as well. I’m willing to invest in being an NP if it becomes available for individuals to host at their residences. This was a major hope of mine for the future. I really like the idea of investing in becoming an NP if the rewards are substantial enough to make it worth while. I also would like to help decentralization in any way I can.

Please keep us (the community) updated!

[bjoern]

@ritvick

• If anyone submits the (“Register a New NP proposal” and other proposals) to add a node to network, on what grounds Dfinity foundation / ICA/ (whoever has majority of following ) approves or rejects it ?

The main purpose of having the registration go through the proposal flow is the validation of the information. That is, to add a new NP, the NNS needs to validate the identity of the node provider. The documentation necessary may differ from case to case. As this is the first time we’ll be adding entirely new node providers post-launch, this probably also deserves some discussion on the forum.

While your assumption 1. is correct, which is that at this moment DFINITY and ICA direct the largest chunk of voting power on the relevant topics, we will try our best to collaborate with the community on the decision.

• I see a lot of nodes are sitting in Awaiting subnet status. And as per my understanding one does not get rewarded unless the node is in a subnet. What is the criteria to get into a subnet ( either old or a new one) ? . Are subnets created automatically or manually for now?

Here we need to differentiate between (a) the current implementation and (b) the intended future mechanism. At this moment, the rewards that are paid out are determined exclusively by the “rewardable nodes” setting in the IC’s registry. That is, the information whether or not the nodes are in a subnet is not yet used for the reward computation; the “rewardable nodes” data structure is the source of truth. That data structure is changed via a proposal to the governance system, which is the last step in the NP onboarding flow.

At the moment, subnets are created and nodes are added/removed from subnets based on proposals, most of which are today generated by DFINITY’s team. Behind the scenes, the team uses tooling that maximizes decentralization based on objective criteria. In the future, we expect to move that code into the NNS so that all decisions on node assignment are performed programmatically by the governance system.

In the future, the remuneration will take more data such as actual subnet membership and node uptime/downtime into account, so the accounting will be a lot more precise.

[SvenF]

Dear community, this is Sven Fischer from DFINITY, part of the Release Management team.

As we are moving into the fourth quarter we would like to give an update on the status of the Node Decentralization activities. To recap, the onboarding of new Node Providers (NPs) was halted due to reprioritization of other activities, amongst others, the Bitcoin Integration. However, other node decentralization activities, like upgrading of existing nodes, and testing of new hardware specifications, have progressed in the meantime.

As of the start of Q4, the status of our decentralization efforts is as follows:

• The upgrading of existing nodes to is in progress and continues until the end of 2022. Storage upgrade will be included in this to accommodate potential growth in storage requirements of the IC network.
• In Q4, we will start onboarding the first new Node Providers. Focus will be on adding nodes of new NPs to the NNS subnet in order to extend our degree of decentralization of the NNS. Furthermore, part of the new nodes will be used to increase the security of the ECDSA subnet used for Bitcoin integration.
• The new NPs will be onboarded using the new hardware specification (Gen2 replica node hardware specification), while we are completing the final tests of the hardware in parallel, specifically on the support of SEV-SNP
• We will start in Q4 with onboarding of 6 new NPs.
• A subset of these new NPs will be asked to onboard independently based on the currently available NP onboarding runbooks (Node Provider Onboarding), in order to show the community that NPs can onboard without intervention or support from the foundation. Technically it is already possible for a NP to onboard, but there are technical practicalities as stated in the above thread that inhibit this, and we want to make the onboarding process a smooth and efficient process.
• Self-declaration: While the new onboarding process provides more independence to NPs, it brings the question how the community can judge which NP proposals to adopt and which ones to reject. The team is preparing a self-declaration template. Through an upcoming NNS motion proposal, we propose to establish this self-declaration as a means for NPs to provide more information to the NNS community, enabling an informed vote on whether to adopt a new NP.
• For the new NPs being onboarded we will follow the existing remuneration schema. That is, rewards are determined exclusively by the “rewardable nodes” setting in the IC’s registry. This might change in the future as the team is looking into taking into account more precise metrics such as node uptime and downtime (see also the previous thread response on Remuneration).

The DFINITY Foundation is currently working together with the community and existing NPs for the above activities. There is a large backlog of NP requests so we expect the decentralization to accelerate over the first two quarters of the next year, with the introduction of runbooks for independent onboarding by new NPs.

Q4 2022

• (A) Continued work on upgrading existing nodes.
• (B) Onboarding of 6 new NPs, with a subset following runbooks for autonomous node deployment. Deployment will be based on the next generation HW configuration that is currently in test.
• (C) Testing of next generation HW for different HW vendors.

Q1 2023

• (C) Testing of next generation HW for different HW vendors.
• (D) Start of autonomous node deployment for new NPs.

Q2/Q3 2023

• (E) NNS-driven DC allocation: The NNS-controlled DC allocation mechanism will be introduced and control all future node additions. With this step and a new node hardware an accelerated network growth is unleashed.
• (F) NP onboarding using NNS Front-end dapp instead of CLI tooling.

Of course, based on community feedback or technical obstacles that we might encounter, the plan might be adjusted. We will continue to update you through the forum on the status of these activities.

[cryptoschindler]

@SvenF

In Q4, we will start onboarding the first new Node Providers.

How are those selected? Is it on a first come first served base from the backlog? If so, is there a way to find out what place one has on the waitlist?

[shayaaa]

We seem to have about ~74 existing NP’s, have the current NP been “fully on boarded” with most of their nodes being operational? Looking at the IC Dashboard there seems to be a lot of gaps. Do you know what could be holding that up?

Am I also correct in understanding that new NP’s will have same rewards as the first round? Although NP rewards are a smaller factor in overall inflation rate, that seems less than ideal.

[SvenF]

Thanks @cryptoschindler, this is a very good question. The team is still discussing on this, but it will most likely take into account different aspects, as we need to take into account decentralization metrics per city, country, region and continent; so it will not only be on first come, first serve basis. As soon as I have more information, I will make sure to update the community in this thread.

[Zane]

Real question is how will they be selected once NNS is responsible for node onboarding

[SvenF]

That is correct @Zane and we will definitely come back on this.

[ritvick]

All this new ob-boarding process looks okay. But to be frank, when does one buy the hardware? Buying the hardware takes a lead time of up to 6 months.
So my questions are:
1.
Whoever wants to become an NP, shall they apply first to become NP? Reach step IX Node Provider Onboarding - Internet Computer Wiki
Because till then it’s all a manual approval process and no one would want to buy such expensive hardware without knowing with 100% certainty what it will take for them to reach that step.
2. Also I understand a Node is not rewarded unless it is in a subnet. Even when a node is onboarded how does the process looks like to get into a subnet? No one would want to have a node onboarded and just pay bills without getting rewarded.

Let’s try to bring more clarity to the community on this.

[SvenF]

Thanks @ritvick for your questions. These are actually very relevant, and the reason for not responding earlier is that we are discussing these questions internally. We are working on clear guidance and proposals in this area that we will share with the community shortly. Thanks again for bringing this up.

[SvenF]

Update November - Node Decentralization

Following up on the roadmap that we posted four weeks ago, I would like to give an update of the decentralization activities since then. Before going into the details, let’s look at the network topology as of today versus last year. A lot of progress has been made with respect to decentralization so far. The table below shows that overall decentralization has increased over the last three quarters on every metric, with the exception of the number of Node Providers (NPs) and Geographies, which is the reason our target for the next year will be to have new Node Providers onboard, in particular in new Geographies.

Storage increase of existing replica nodes

To recap, Node Providers are currently upgrading their existing configuration, allowing these Node Providers to independently manage their nodes, while at the same time increasing the storage capacity of the nodes from 3.2 TB to 32 TB. This upgrade is in full progress, with Asia nodes being 100% completed, European nodes 94 % (1 Node Provider left) and US nodes 44% completed (9 Node Providers left). We expect the upgrade of existing Node Providers to be fully completed by the end of the year.

When the upgrade of existing Node Providers is completed, the Dfinity team that is consulting these Node Providers can start to focus on the autonomous onboarding process for new Node Providers.

Testing of 2nd Generation Hardware

The minimum requirements for the next generation of replica node hardware (Gen2HW) have been published on the Internet Computer wiki, as well as three specific configurations of HW that have been validated by the Dfinity team. Validation of other HW specifications is in progress, and for discussion please refer to the separate Forum post New Hardware Specification.

Testing of the fully autonomous NP onboarding process

The new NP onboarding process consists of two parts, covered by two separate runbooks: the onboarding of a new Data Center (DC) and the registration and acceptance as a new Node Provider by the NNS. The latter runbook was already available on the wiki. The DC runbook is completed as well, and is currently being verified internally. The next step will be to test both the NP registration runbook and the DC setup in combination with Gen2HW. A brave potential new NP has agreed to test this process in November and December. The progress of these tests will mainly be determined by the lead time for procuring specific Gen2HW.

Remuneration

Cycle pricing and node provider rewards (remuneration) require refinement as the network grows. More concretely, we propose the following changes with respect to node provider rewards:

• Today’s Genesis remuneration model: In early 2021, NPs had to commit to a not-yet-launched network. To compensate for the risk taken, the original genesis remuneration model includes significant profit margins. This model does not cover all of the storage extensions described above, so the foundation has been covering a portion of those costs, while the storage was not benefitting the network.

• Interim Gen2HW-ready remuneration model: While a more general remuneration overhaul is required and needs to be discussed mid-term (see next bullet point), an interim update is required as the storage upgrade comes to a completion and NPs using Gen2HW are onboarded. We propose the remuneration to be take into account two factors. First, HW costs have risen and need to be compensated. However, having a running IC network reduces the risk of entry for new NPs, hence a lower risk premium is being included in the reward. Taking into account these above factors, the interim remuneration is lower than the Genesys remuneration model. The wiki page (Node Provider Remuneration) shows the draft of the interim remuneration model that we propose to use. It is based on currently available hardware prices and average rates for ICP and XDR. As shown from the table, the premium multiplier for remuneration will lower from 2.5 to 1.9 to take into account the reduced risk premium for joining the IC network. We look forward to your feedback in this forum thread, plan to incorporate it into the interim remuneration model and then submit the respective proposal to the NNS.

• Future remuneration model: We acknowledge that the interim remuneration update does not address all open questions such as how to incentivize decentralization through an adaptive remuneration. A more general update needs careful preparation and a thorough discussion with the community. We plan to start that discussion during the next few weeks.

Node provider self-declaration

As already asked by Jordan Last about a year ago, the autonomous NP onboarding process requires incentives for NPs and node operators to act in the interest of the IC. Moreover, the community needs sufficient information in order to be able to decide whether to adopt or reject a new NP. As sketched by Dom on Medium, we plan to discuss and propose a self-declaration for NPs. We will present that in a separate thread shortly. Stay tuned.

Way forward for onboarding

With the runbooks tested and with a proposal for interim remuneration, we can start onboarding new node providers in a controlled way. Currently, the onboarding process requires sound technical knowledge, which is something we are aiming to reduce in the coming weeks and months. The steps to follow are roughly as follows:

• NP to prepare self-declaration (separate thread to follow shortly)
• NNS registration through proposals (see Wiki instructions including all prerequisites)
  • Register as an NP (including pointer to self-registration)
  • Create a new data center record (if that does not exist already)
  • Create a node operator record (if that does not exist already)
• NP orders hardware (see verified 2nd gen replica node configurations on the wiki)
• NP/node operator setup DC, install nodes, set up the reward configuration and have them join the network.

Lead time for ordering hardware can be up to several weeks or even months, thus the above process will take time. Most of the work is with the node providers: preparing documents for self-declaration, submitting proposals, purchasing hardware, setting up the nodes, etc. As outlined in earlier posts, we plan to provide a better user experience for some of these activities, especially for submitting proposals and changing the NP profile. For now, however, NPs should be comfortably working with a console.

As explained in the introduction of this post and also motivated in the Medium article Let’s Grow the Internet Computer Network, the IC network needs to grow. In particular, the network needs

• more, i.e. new, node providers and
• nodes in underrepresented geographies, such as Asia, Australia, and Africa in order to improve decentralization.

Call for action

In summary, we encourage

• new node providers
• who are technically savvy and not afraid to be among the first to run the autonomous onboarding process
• and plan to contribute nodes in underrepresented geographies

to start reviewing the hardware specifications, to familiarize themselves with the runbooks, and to stay tuned for the upcoming self-declaration discussion.

We propose to start with small batches of node machines (e.g. 5 per new node provider and location). As the process matures based on the feedback of these early adopters, we expect the numbers to grow more rapidly.

We look forward to feedback from the community, especially from potential future node providers.

[Mr_Burkes]

QQ: What defenses does the network have against bad node providers?

What if we onboard a bad node provider who only violates the protocol? How will people on the NNS know if a node is bad? Are node rewards slashed?

[plasmex]

From what I’ve gathered there is an SLA as a deterrent for downtime.

As far as validating node specs beyond simply being hard down, not completely sure.

[jzxchiang]

Thank you for sharing the plan.

A few questions:

1. The WIP Gen2 renumeration model assumes an annual inflation rate of 10% when estimating operating expenses over the next 4 years. That seems overly pessimistic. According to the IMF, global inflation will be 8.8% in 2022 but decline to 6.5% in 2023 and to 4.1% in 2024. Also, developed countries typically experience less inflation than developing countries; I wonder if a more fine-grained inflation model makes sense here.

2. On the topic of OpEx, I am curious why Asia OpEx is nearly twice USA OpEx. The wiki says those estimates are based on numbers shared by current NPs. Wouldn’t NPs have an incentive to inflate those numbers to receive larger rewards? What checks and balances do we have to prevent NPs from lying? Could the NNS leverage the new HTTPS outcall feature to query external sources without relying solely on NP claims?

3. It is difficult for a layman like myself to assess whether a 90% profit margin is appropriate or not. How does this compare with other blockchains, e.g. validators on Ethereum? How does this compare with cloud providers? Should the margin be automatically adjusted based on some formula in the future?

4. Somewhat unrelated, but this wiki page says “Renumeration of node operators” instead of “Renumeration of node providers”. I’m assuming that is a typo.

Thank you!

[Mr_Burkes]

I’m bumping this because this is an important question. How can we ever add non-Dfinity-approved nodes if we don’t have a guarantee of protocol disincentives?

[memetics]

Add badlands to the roadmap!!!

[SvenF]

@Mr_Burkes @plasmex these are very important questions indeed. we can automatically check some elements in advance (see also this other forum post: Proposal for Node Provider Self-declaration) but there is of course still to potential for violating the protocol. There are two features we are discussing and preparing internally, work in progress: an automated reward system in the NNS that includes penalty for policy violation, and observability/monitoring of NP performance. Both are on the roadmap and being worked on.