The State and Direction of Decentralization & Nodes on the Internet Computer

I partially agree, but the hw and data center rent aren’t necessarily lost, you could make a deal with someone else and cover your losses or even make a profit, staked ICP can be slashed, so in the exact moment the node provider doesnt behave correctly he permanently lost money.

3 Likes

This is also the GRT model. You do stake GRT (100k grt) in addition to node. (& you get slashed for degraded performance; wrong results in grt case). The GRT node costs are similar to IC for most performant GRT node. Of course in IC case, there’s one type of node at this point.

2 Likes

I believe that the DfinityNodes project is on the right path to bring more decentralisation by crowdfunding nodes through their coming DAO. If all checks out on their end it would be beneficial for everyone if they got the right support and backing by Dfinity. There will always be minimum amount of hassle for anyone involved if it goes through one source and what better than a DAO that acts as a node operator backed by thousands of shareholders.

Just my 2 satoshis

5 Likes

I agree that slashing could help the node operator keep skin in the game for good behavior.

My question is can we distinguish between “bad behavior” and unexpected interruptions in service (such as a power or internet outage to the node provider)? We could even imagine an organized group putting pressure on a node by threatening to cut off its power or internet and thereby lose its stake. We would want nodes to be resilient against these kinds of threats.

5 Likes

@Zane @lastmjs Please, consider also (there are already → correct to → there will be) measures in place to counter bad behavior / bad performance, that is a penalty on the node rewards: penalty table .

On top of that, a single core node with storage upgrade costs ~30k $USD. Now, if we would propose to add an extra layer of investment (locked ICP associated with the node), how much would it be? 10k $USD (to make some sense)?

So, we would increase the barrier to get a new node up and running, from 30k $USD to 40k $USD.

On top of that, you can hardly get colocation in a good Data Center (with a good structure) if you want to rent a single 1U in a rack. Usually, they offer you 12 to 24U, not less than that. Meaning, a node provider needs at least 3-4U to start getting into the business.

My point is: There is already a huge financial barrier for someone to become a node provider. We are talking about ~100k $USD of the initial investment.

Do we really want to increase this barrier even more?

The community keeps saying “anyone can become a node provider” … Currently, anyone with > 100k $USD to put into hardware.

For me, it feels like locking ICP into nodes makes it even more difficult to expand the network into millions of nodes. The requirement for top-performance hardware is already a strong incentive (also, a strong barrier for new node providers); and it is already a locking as noted by @diegop .

7 Likes

Update:

Aas part of the follow-up to this post: Motion Proposals on Long Term R&D Plans (Please read this post for context).

We have begun posting Long term R&D project 1-pagers. This project will have the 1-pager in this thread.

We should add legal fees to the cost of being a node provider since that burden is being placed on them now.

1 Like

@llbrunoll
This is Luis Mompo Handen from DFINITY. I would like to add some slight corrections to your post.

Penalties are not yet in place. There two things that the IC needs to provide to do so: 1) automated reward payouts (roadmap coming soon) and 2) decentralized node addition (roadmap published).
With 1) penalties will be calculated by the NNS in the same way rewards are and with 2) node providers can take the full responsibilities over their nodes.

Node providers have currently three types of nodes. From about 8K$ to 30K$ whereby the 30K$ type is a minority of the network and is meant for storage subnets that are still in development. On the running costs we also have three different types depending on the number of nodes per DC and rack (6,14 and 28). I think long term there will be many types of nodes with various costs and therefore various rewards.

The deposit neuron for node providers is an idea that IMO will need more discussions in the node provider community. I would like to be careful with things that would overregulate a market for no reason. As long node providers are supporting the IC in the way they are currently doing and seeing a long list of interested node providers I don’t see a reason for such a deposit. I think we all agree that growing the network is the best we can do to ensure the resilience of the IC. Getting accepted as a node provider by the community and thereby by the NNS should be a highly valuable accreditation and of course a honor. Ideally this would always make any kind of regulation obsolete.

11 Likes

For context… Luis above is the team lead at Dfinity for node provider efforts. He works with @yvonneanne (researcher) and @samuelburri (vp of engineering).

1 Like

Hi @Luis , thank you for your comments, brings a lot of extra information. I just added a note there regarding penalties (that are not yet implemented). I have tried to understand details about ‘how to become a node provider’ , talked to local DataCenters to check about colocation, and understand about operation costs. Information about node providers is very sparse, I could only grasp what is happening, by collecting info from posts here and there, and by talking to one node provider (who gave me some info, but not a lot, due to “confidentiality requests enforced by contract” - that is what I was told) … For example, this is the first time I hear that there are three configurations (6, 14, and 28 nodes) with different associated costs. Could you give us more info about those configurations and the hardware cost involved? Thank you!

Thanks for the infos, right now nodes are paid a fixed amount regardless of how many cycles they process, but I’ve read many people claim in the future it will be based on cycles, is that true or a fake rumour?

3 Likes

Is the cost of the core node machine destroying the network too low?
According to the ICA document, the core node machine can also be used as a general-purpose server.

The result of node machines destroying the network is nothing more than losing costs. They can also go to the second-hand market to sell these general-purpose servers after destroying the network. Will this create the possibility of large-scale evil?

2 Likes

In addition to the short- and mid-term items listed for the first quarters of 2022, the following topics will be addressed in the next few years. The ultimate goal for all node provider-related improvements is sustainable network growth. While the short- and mid-term items focus on improving the autonomy of node providers, the items listed below aim at operational improvements and automation, a prerequisite for an ever-growing network.
Note that work described in the Motion proposals on Trusted Execution environments, Secure OS, E2E Security, Malicious parties, Scalability and Tokenomics is required to be integrated into the release and operational processes to achieve these goals. We will not describe or discuss them here in detail though and suggest questions regarding them to be asked in the corresponding forum threads.

Operational Maturity

  • Decentralized monitoring: Currently, the health of the nodes is measured by collecting and analysing logs and metrics on a third-party cluster. When deviating from the expected indicator values, the respective node providers and data centers are notified manually to fix the situation. In the next generation of the internet computer protocol, the monitoring tasks will be carried out by the nodes of the network themselves, in a fully automated fashion. To this end, the architecture of the nodes and their components needs to be extended. the interaction between the different types of nodes and subnets needs to be adapted and . A mechanism to deal with faults within and outside the control of the node owners and data centers must be explored. In particular, it should be possible for any party to collect information on the health and contributions of any node without additional trust assumptions.

  • Decentralized backup: To ensure that even in the presence of a deadly bug in the node software the content of subnets is not lost, a backup mechanism shall be implemented to collect the state and the messages sent to a subnet off-chain. Using subnet recovery, the state of the subnet can be re-generated. In order to do this in a decentralized manner, mechanisms trading off replay speed, memory complexity, and privacy concerns have to be devised. Furthermore, the backup mechanism must be integrated in the NNS-driven

  • Decentralized boundary nodes: As elaborated in a separate motion proposal, boundary nodes will become fully NNS-managed and use the same update mechanisms and operating system as regular nodes. This will allow the inclusion of boundary nodes in the new node provider deployment process with slight variations in the configuration process.

  • Extended node remuneration: The previous items will lead to refinements and improvements of the node remuneration process. Insights gained through the decentralized monitoring will allow automated adjustments to payments based on the performance of nodes, or rather the lack thereof. Backup services and boundary nodes will be new categories to be included in the remuneration process.

Automation

  • Automated node and data center allocation and rewards: In a future version of the Internet Computer protocol a lot more tasks will be automated and executed by NNS canisters. In particular, the computation of how much rewards a node in a specific location should be granted, will be taking the current local costs for electricity, wages, regulations and risks into account and combine this with the measurements obtained from the decentralized monitoring infrastructure. The future versions of the NNS will be designed to reflect the dynamics of compute and storage demand and supply, integrate decentralization metrics and balance the amount of tokens in circulation.

  • Automated subnet creation and healing: With a continuously increasing number of nodes and subnets, it will become untenable to manually propose removal of unhealthy nodes and addition of new ones. Instead, the NNS will be extended to propose and execute node replacements based on data collected through decentralized monitoring.

Note: In the upcoming Decentralized Node Management motion proposal (see Motion Proposals on Long Term R&D Plans) we plan to combine the initial post in this thread, these additional points and some reactions and adjustments based on the conversation here.

5 Likes

For completeness, we decided to also publish the motion proposal that we plan to submit on Monday. It combines the first post in this thread the the extension posted earlier today.

Decentralized Node Management

Summary

The Internet Computer (IC) is formed by standardized “node machines”. These machines are owned by independent node providers and installed within independent data centers. This motion proposal describes the technical and organizational advances required to guarantee a continued, sustainable network growth of the IC.

1. Objective

The objective is threefold

  • Decentralization: empower node providers to setup, monitor and maintain nodes independently.
  • Scalability: remove technical and operational bottlenecks that hinder the network to grow to millions of nodes.
  • Automation: remove errors and speed-up repetitive tasks to lower risks and reduce resources required for the ever-growing IC network

2. Background

Compared with other blockchains, the IC has stronger requirements on the homogeneity of the node machines due to the fact that most of the machines’ resources are dedicated to perform useful work (e.g. executing smart contracts, participating in threshold cryptography) that needs to be performed by all nodes of a given subnet blockchain.

The IC is designed to host mass market applications completely on chain. Thus the state of canister smart contracts on the IC can be much larger than the state of smart contracts on other blockchains. The nodes also need significant network capacity for managing and synchronizing that state. To meet the requirements of extremely high availability and high throughput, the IC nodes are hosted in data centers (DCs). To achieve decentralization, the nodes are owned and the DCs are operated by independent parties. Moreover, the DCs are spread across the globe. Due to the need for homogeneity and capacity, the Internet Computer Association (ICA) defines the hardware to be used for node machines. The network is expected to grow to millions of nodes running from thousands of DCs in the coming years. As decentralization is key to the mission of the IC, many new, independent node providers (NPs) will be needed to support that growth. The current NP onboarding process, node deployment and management procedures are not ready to sustain this network growth mid- and long-term.

3. Why is this important?

A successful IC attracts thousands of developers who are continuously deploying millions of canister smart contracts. In order to sustain this growth, the number of nodes and subnets must steadily increase. However, this growth can only be sustained if the operational processes are further automated, ready to scale – all in a decentralized fashion.

4. Topics under this project

Specifically, this proposal includes the following research and development directions. The first topics are reasonably well scoped and work has started. The second set of topics is to be tackled mid- and long/term.

Short-term

  • Autonomous node deployment and operation: Empower node providers to install and maintain nodes without any support from the ICA.
  • Independent node provider registration: New node providers shall be able to register directly in the NNS frontend dapp.They will be able to submit a proposal to become a new node provider, without the support of the ICA.
  • NNS-driven DC allocation: Node providers can participate in an NNS-controlled mechanism to elect new data centers. This process shall incentivise the addition of new DCs that further decentralize the IC’s network. Furthermore, it balances inflationary and deflationary forces.
  • Availability of node hardware: A new generation of ICA-specified node hardware is planned that shall guarantee global availability and provide a better choice between hardware providers. Furthermore, the next generation shall include the SEV-SNP technology to further improve the security of nodes.

Mid- and Long-term

  • Decentralized monitoring: Currently, the health of the nodes is measured by collecting and analysing logs and metrics on a third-party cluster. When deviating from the expected indicator values, the respective node providers and data centers are notified manually to fix the situation. In the next generation of the Internet Computer protocol, the monitoring tasks will be carried out by the nodes of the network themselves, in a fully automated fashion. To this end, the architecture and the protocol of the nodes and their components require extensions. In particular, it should be possible for any party to collect information on the health and contributions of any node without additional trust assumptions.

  • Decentralized backup: To ensure that even in the presence of a deadly bug in the node software the content of subnets is not lost, a backup mechanism shall be implemented to collect the state and the messages sent to a subnet off-chain. Using disaster recovery, the state of the subnet can be re-generated. In order to do this in a decentralized manner, mechanisms trading off replay speed, memory complexity, and privacy concerns have to be devised. Furthermore, the backup mechanism must be integrated in the governance protocol, including the assignment of which nodes are responsible for the backup of which subnets and their monitoring.

  • Decentralized boundary nodes: As elaborated in a separate motion proposal, boundary nodes will become fully NNS-managed and use the same update mechanisms and operating system as regular nodes. This will allow the inclusion of boundary nodes in the new node provider deployment process with slight variations in the configuration process.

  • Extended node remuneration: The previous items will lead to refinements and improvements of the node remuneration process. Insights gained through the decentralized monitoring will allow automated adjustments to payments based on the performance of nodes, or rather the lack thereof. Backup services and boundary nodes will be new categories to be included in the remuneration process.

  • Automated node and data center allocation and rewards: In a future version of the internet computer protocol a lot more tasks will be automated and executed by NNS canisters. In particular, the computation of how much rewards a node in a specific location should be granted, will be taking the current local costs for electricity, wages, regulations and risks into account and combine this with the measurements obtained from the decentralized monitoring infrastructure. The future versions of the NNS will be designed to reflect the dynamics of compute and storage demand and supply, integrate decentralization metrics and balance the amount of tokens in circulation.

  • Automated subnet creation and healing: With a continuously increasing number of nodes and subnets, it will become untenable to manually propose removal of unhealthy nodes and addition of new ones. Instead, the NNS will be extended to propose and execute node replacements and subnet creation based on data collected through decentralized monitoring.

Note that work described in the Motion proposals on Trusted Execution environments, Secure OS, E2E Security, Malicious parties, Scalability and Tokenomics is required to be integrated into the release and operational processes to achieve these goals. We will not describe or discuss them here in detail though and suggest questions regarding them to be asked in the corresponding forum threads.

5. Key milestones

The following milestones are indicative and their scope may change as the work proceeds.

  • (M1) Decentralized node deployments: This milestone subsumes the first short-term items and enables node providers to independently onboard and deploy their nodes.
  • (M2) Redeployment of existing nodes using new deployment mechanism
  • (M3) NNS-managed DC allocation: New DCs are selected by the NNS, possibly using an auction mechanism.
  • (M4) Operational maturity, including decentralized backup services and monitoring
  • (M5) Metrics-based remuneration: a refinement of the node provider remuneration process, taking the data collected through decentralized monitoring into account.
  • (M6) Automation: this is less of a milestone but rather a continuous effort.

Note: Boundary node milestones are described in a separate motion proposal.

6. People involved

Discussion leads: @Luis @yvonneanne @samuelburri

7. Why the DFINITY Foundation should make this a long-running R&D project

The design, implementation and rollout of an enhanced node management requires changes to many layers of the IC stack but also a close collaboration with various stakeholders, such as the node providers. While community members may have an excellent understanding of individual components or aspects, it’s unlikely that the community will drive this change end-to-end without a major contribution of the DFINITY foundation. For that reason and given the importance of the topic for a sustainable network growth, the foundation is determined to make significant resource investments in this project.

8. Skills and Expertise necessary to accomplish this

The problems described above require the cooperation of hardware, data center, and networking experts as well as software engineers, to design, review, and implement the prospective solutions. Specifically, experts from the following fields are necessary:

  • Server design and configuration
  • Networked systems
  • Network management
  • Network security
  • Distributed systems
  • Economics

This project would require both researchers and software engineers with expertise in the above-mentioned fields.

9. Open research questions

  • What are the relevant metrics to derive information about the health and correct operation of nodes? How can this information be collected reliably in a trustless fashion despite failures?
  • How should subnet backup information be distributed, stored and retrieved from nodes while respecting privacy and guaranteeing efficient recovery despite Byzantine node faults?
  • How should the rewards for boundary nodes and backup nodes be determined in relationship to replica nodes? Should there be a role shuffling scheme and if yes how should it be designed to balance fairness and efficiency while providing maximum security?
  • How can the computation and implementation of penalties and rewards be automated in a fair and incentive-compatible way with minimal implementation effort?
  • How to migrate from the current deployment of three different node types to a unified and more flexible deployment fully governed by the NNS?
  • How to design allocation mechanisms that provide incentives for network growth maximizing decentralization along multiple dimensions? How to balance dynamism in local costs, risks and regulations with the need for long-term planning and the supply and demand of ICP overall?
  • How to exploit the information about node health and subnet usage for the automatic derivation of suitable thresholds for node replacements and subnet creation?

10. Examples where the community can integrate into project

Node providers are the key stakeholders for the plans presented in this proposal. We would greatly appreciate their opinion on how to prioritize the work items listed above and whether we may have missed some important aspects. We plan to rollout first DCs with the new deployment mechanism in Q1 2022 and look forward to tangible feedback from these first trials. We plan to keep the community posted on this topic on a regular basis.

11. What we are asking the community

The forum discussion around the roadmap for node management has attracted a lot of interest and contributions from the community during the past days. We have included feedback that we have received so far in this summary. DFINITY engineers and researchers are looking forward to more inputs and discussions once this motion proposal is submitted.

8 Likes

NNS Motion is live!

https://dashboard.internetcomputer.org/proposal/35670

Thank you for the detailed post. This is surely one of the most important items on the IC roadmap.

NNS-driven DC allocation: Node providers can participate in an NNS-controlled mechanism to elect new data centers.

Can you explain what it means to elect data centers? I thought the NNS’s job was to add new nodes to subnets, which are run by registered node providers. Where does the concept of a data center come into play? I thought that was more of an implementation detail for node providers where they choose to run their nodes (except for the geographic location).

1 Like

Is there a list of potential new node providers who will be onboarded using (C) in Q1 2022?
I would love to be on that list!

I’m currently negotiating an option to setup nodes in a new datacenter in Ethiopia.
Please let me know where I can indicate my interest to be considered for this early adopter phase.

2 Likes

Hello everyone - Cole from the DFINITYNodes Team here!
I just wanted to introduce our project & ideas to the DFINITY Team and Community, to get some feedback, and hopefully work together to bring this project to reality.

DFINITYNodes is a community solution to the inaccessibility of Nodes on the Internet Computer to the average user. We plan to solve this issue by utilizing a DAO to crowd-fund for Nodes through NFTs, allowing users to get involved at a low entry cost. These NFTs are referred to as “Node Shares”, which are stakeable within the DFINITYNodes DAO to participate in governance and reap Node Rewards for participation.

For a deeper dive into the mechanics of DFINITYNodes, please reference the DFINITYNodes Medium, where you can find our Whitepaper & Introduction.

7 Likes

I’m happy to be helpful and I guess many interested NPs are following this thread. Your feedback is helping us to find out what needs clarification, what needs to be discussed with the community.

This roadmap post wants to inform about the current state and the roadmap for the next months and thereby it covers the onboarding of new NPs with some “modified Gen1 hardware” in phase 2 (see graphic). This 2nd phase isn’t meant to support the growth of the network as the planned growth is fully achievable with the 1st generation of NPs (see total node count on the bottom of the timeline) but to prove that a decentralized onboarding process of new NPs and DCs implemented into the NNS UI is actually working. The NNS driven data centre auctions in the 3rd phase are depending on this NNS UI workflow and its verification can’t be done with the 1st generation NPs because they are already onboarded, means registered in the NNS.

At the time at this post was written we had three slightly different and very detailed sequence diagrams showing this process. We chose a wording that matched with all three variants. This is not because we want to hide something but because we admit that there is a lot of internal adjustments needed to decide what is feasible within the first year of the IC.
The list of interested node providers is long. The new node providers that will be added this year to verify the decentralized onboarding process will very likely be chosen out of this list. From there on all technical prerequisites are given for decentralized growth of the IC. What’s then missing is a growth strategy that leads from current subsidized pricing model to a utilisation based pricing model.
We are preparing a forum post that wants to explain the current subsidized model and start the discussion with the community about a healthy growth of the IC and therefore show how new node providers can join the network.

3 Likes

Hi Cole and welcome to the node provider community.

I read your medium post about your project and I think that’s a really good idea to contribute to the growth of the network. I see a little limitation that comes along with the nature of the IC. Our decentralisation algorithm ensures that no single entity isn’t able to manage/access a critical majority of the nodes in a subnet. Your project must be seen a single node provider due to the access level that is given by the fact that you are running the nodes for your investors. This somehow limits the growth of your project to the growth of the IC but still works and still allows sharing the investment with your users.
Beside that I would see the same prerequisites for your project than for any other node provider. As explained in my response to @llbrunoll we need to wait until everything is given to start onboarding new node providers.

3 Likes