The State and Direction of Decentralization & Nodes on the Internet Computer

[diegop]

Introduction and Vision

The Internet Computer (IC) is formed by standardized “node machines” from independent owners and installed within independent Data Centers (DCs). Compared with other blockchains, the IC has stronger requirements on the homogeneity of the nodes due to the fact that most of the node resources are dedicated to perform useful work (e.g. executing smart contracts, participating in threshold cryptography) that needs to be performed by all nodes of a given subnet blockchain.

The IC is designed to host mass market applications completely on chain. Thus the state of canister smart contracts on the IC can be much larger than on other blockchains. The nodes also need significant network capacity for managing and synchronizing that state. To meet the requirements of extremely high availability and high throughput, the IC nodes are hosted in DCs. To achieve decentralization, the nodes are owned and the DCs are operated by independent parties. Moreover, the DCs are spread across the globe. Due to this need for homogeneity and capacity, the Internet Computer Association (ICA) publishes specifications for hardware of IC nodes as well as the service levels of DCs.

The IC is designed for anyone to be able to become a node provider (NP). The governance system of the IC, the Network Nervous System (NNS), scales the IC’s capacity by adding more nodes. The requirement of additional nodes depends on the desired capacity (and location requirements) of the IC at a given point in time. The governance system is designed to remove nodes that provide sub-par performance, it is thereby recommended to follow the specifications published by the ICA. The IC constantly scales out its compute capacity by adding new nodes to the network such that it never runs out of capacity.

The network is expected to grow to millions of nodes running from thousands of DCs in the coming years. As decentralization is key to the mission of the IC, many new, independent node providers will be needed to support that growth.

The Network Nervous System (NNS)

The NNS is the on-chain governance system of the IC. Holders of the Internet Computer’s ICP utility tokens can stake their ICP tokens in neurons to participate in governance and contribute to decision-making, such as voting to determine whether or not a new collection of nodes (also called a subnet) should be added to the network. It has many functions, but the most relevant ones to node decentralization are:

• Maximize the decentralization of the node providership and control.
• Determine the number of nodes required to match the IC’s compute/storage demand while keeping the inflation in check (not minting too much ICP to pay for node provider rewards).
• Set the cost of compute and storage operations for developers.
• Remove faulty or slow nodes from the IC.
• Upgrade the replica software running in the network.

How to become a node provider

Here are the envisioned steps to become a node provider for the IC:

• Using the NNS frontend dapp, everybody can register to become a node provider. Registrations are approved by means of NNS proposals, voted on by the IC community.
• The ICA publishes hardware specifications for nodes. Node providers can order machines fulfilling these specifications from different hardware vendors.
• Node providers propose new data centers using the NNS frontend dapp. The NNS approves data center proposals depending on how they further improve the decentralization of the IC. For example, data centers in underrepresented geographical locations are preferred over those in locations with many existing nodes.
• Data center operators deploy node machines independently, using the opensourced node software and configuration data retrieved through the NNS frontend dapp.
• The remuneration for running a node starts as soon as the node has joined a subnet.

Responsibilities of node providers

A node provider performs the following operations:

• Make proposals to become a node provider and to add new data centers.
• Purchase nodes that satisfy the ICA hardware specifications
• Rack and stack the node hardware and install the node software. This work may be outsourced to a 3rd party
• Ensure a guaranteed uptime of the nodes

How the NNS manages inflation and deflation

Canister smart contracts running on the Internet Computer blockchain are fueled by “cycles” which in turn are generated by burning ICP thereby causing deflation. As the number of smart contracts on the Internet Computer grows, so does the need for adding additional nodes.

Each node receives a flat monthly reward calculated in fiat and paid in the form of ICP tokens. This model ensures that the cost that the network bears for running a node is always constant in fiat terms. By keeping a predictable cost the network is able to ensure that the developers pay a flat fee for each operation on the Internet Computer independent of the volatility of ICP. The reward varies slightly based on the geographical location of the node to ensure the IC is able to achieve a wider geographical distribution of nodes. Each node provider continues to receive rewards as long as they guarantee the right quality of service. The rewards are paid by minting new ICP tokens, causing inflation.

The NNS has an algorithmic way to react to the demand & supply needs of the IC to keep the community-desired inflation-deflation range in check. The community updates or interacts with the algorithm via NNS proposals.

Glossary

• A node is the hardware that hosts and runs an instance of the IC software stack, known as the “replica.” In other blockchains, this may be referred to as a “client”.
• A node provider (NP) is a person or entity that owns nodes that are part of the IC. Each node provider can host nodes in different data centers, possibly in different countries. You can see examples of owners here: https://dashboard.internetcomputer.org/owners
  In the NNS, node providers are referenced by a principal. A Principal corresponds to the person or entity who receives the rewards stemming from node participation. An example of a Principal is: 7k7b7-4pzhf-aivy6-y654t-uqyup-2auiz-ew2cm-4qkl4-nsl4v-bul5k-5qe.
• A data center (DC) is a physical location at which nodes reside. Everyware AG, Zurich is an example of a data center where IC nodes reside.
• A DC operator (DCOP) is a person or entity responsible for installing and maintaining nodes within a DC. If the NP owns nodes in multiple DCs, multiple DCOPs may be working for the NP. A NP can also act as a DCOP or outsource the DCOP work to a third party.

The state of the IC since Genesis

Above, we describe the vision for the onboarding of new node providers. In the following, we describe the status of the onboarding process. In the next section, we then provide a roadmap leading from the current to the envisioned state.

Prior to the Genesis of the network, the DFINITY foundation worked to build a baseline reserve capacity for the initial subnets. At Genesis, the IC was composed of 130 nodes.

Immediately after Genesis, the NNS ingested a substantial number of nodes to create a baseline of capacity. As of November 2021, the IC is composed of 451 nodes, some of which are spare capacity that is added continuously to the IC and a few are ready to be used in case of incidents.

Nodes are paid in ICP minted by the NNS as described in the vision.

The IC experienced a strong interest from node providers ahead of the network’s launch in May 2021. In anticipation of the new onboarding process and to avoid very long waiting times, the ICA paused further onboarding in Summer 2021. As of November 2021, there is a backlog of 890 nodes waiting to join the IC. These nodes are owned by node providers who have already been onboarded. This backlog guarantees a steady growth of the network while the onboarding process is further improved and finally opened to future node providers.

Current blockers for anyone to contribute nodes to the IC

The following blockers must be addressed before the IC’s network growth can be further accelerated:

• (A) Reproducible builds: The community, including node providers, shall be able to build the replica software in a deterministic manner. This enables the community to verify that IC update proposals indeed correspond to the source code version specified in the NNS proposal.
• (B) Autonomous node deployment and operation: Empower node providers to install and maintain nodes without any support from the ICA.
• (C) Independent node provider registration: New node providers shall be able to register directly in the NNS frontend dapp.They will be able to submit a proposal to become a new node provider, without the support of the ICA.
• (D) NNS-driven DC allocation: Node providers can participate in NNS-controlled mechanism to elect new data centers. This process shall incentivise the addition of new DCs that further decentralize the IC’s network. Furthermore, it balances inflationary and deflationary forces.
• (E) Availability of node hardware: A new generation of ICA-specified node hardware is planned that shall guarantee global availability and provide a better choice between hardware providers. Furthermore, the next generation shall include the SEV-SNP technology to further improve the security of nodes.

Roadmap

The DFINITY Foundation is currently working together with the community and existing node providers to remove the blockers described above. We estimate to complete this process in Q3 2022 with visible progress during intermediate quarters. Based on community feedback or technical obstacles that we might encounter, the plan might be adjusted.

Q4 2021

• (A) Reproducible builds: Reproducible replica software builds are planned to be available to the public in November 2021. This initiative has its own separate communication because it touches many parts.

Q1 2022

• (B) Autonomous node deployment and operation: In early 2022, a new deployment process will be introduced based on the reproducible builds (A). Over the subsequent weeks, existing node providers will be asked to update their deployments using the new process.
• (C) Independent node provider registration: In early 2022, future node providers will be able to register via the NNS frontend dapp. In parallel with the onboarding of the currently backlogged nodes, a limited number of new DCs and nodes will be added to the network using the new deployment method (B).
• In order to get started with autonomous node deployments (B) before the NNS-driven DC allocation is in place (D), a small set of new node providers will be onboarded using (C).

Q2/Q3 2022

• (D) NNS-driven DC allocation: The NNS-controlled DC allocation mechanism will be introduced and control all future node additions. With this step and a new node hardware, see (F), an accelerated network growth is unleashed.
• (E) Availability of node hardware: Different hardware options are currently being investigated. This blocker might be overcome in two steps
  1. Minor extensions to the Genesis hardware setup to overcome current availability limitations.
  2. Larger extensions that incorporate additional hardware technologies such as SEV-SNP.

From now on, updates to this timeline will be updated via technical blog, developer forums, and other social platforms.

[diegop]

[diegop]

The main folks involved are @Luis, @samuelburri, @yvonneanne among others.

[lastmjs]

This is amazing, but somewhat terrifying. What’s been nice so far is that as a community we haven’t had to vet node operators ourselves. I have been assuming that the ICA/DFINITY has been working very very closely to ensure a trustworthy set of initial node operators.

At the end of this process, the flood gates will open and the community will be in charge of ensuring node operators are honest. Because incentives are much more different on the IC than on Bitcoin or Ethereum, I’m concerned with how well they will work.

How will we ensure that node operators are incentivized to be honest? And how will we verify their honesty?

Is there any type of staking?

Can we force secure enclave attestations of the replica, combining it with consensus, so that we can verify the replica has not been modified?

Should we look into more automatic ways of compensating node operators? A fixed fee determined by vote seems much more subject to manipulation than a fee model that pays node operators in cycles, in addition to burning cycles.

I just want us to set ourselves up very well for a much more permissionless system, and I’m not sure we’re there yet.

[mparikh]

@mompo @samuelburri @yvonneanne @diegop

Exciting times to be sure. A couple of high level questions:

Q1. Would testnet/s be also under the same purview?

Q2. How would the concept of partial node ownership (i.e. dfinitynodes) be blended in into the node provisioning?

– the specific question is that would dfinitynodes be treated like any other node provider…so others who want provide similar capability like dfinitynode would need to roll their dao? Obviously dfinitynodes would use their dao to gain an competitive advantage; unless that dao is open sourced.

[diegop]

Very good questions, Jordan.

The simple truth is that this is the high-level vision, many nitty-gritty details need to be worked out together with the community to make it a reality. The original version of this post was 5x longer, but we opted for simplicity and more dialogue rather than producing tomes, so I am not surprised there are questions.

We have been considering these issues, haven’t written much about them yet (will be tackled in different motion proposals we are working on).

What I consider to be more baked:

• Locking some amount of ICP is one of the mechanisms we’re considering to incentivize node providers, and the hardware is actually of non-negligible cost, so you can view this as staking as well.
• We are working on Trusted execution enhanced nodes, incl remote attestation, there will be a separate motion proposal on it (and update coming this week). See: AMD SEV Virtual Machine Support and more coming soon.

What I expect will iterate more through community feedback:

• I expect the AMOUNT of ICP to be something that is calibrated and goes through community discussions.
• Different models for node operator remuneration mechanisms have been discussed internally (pros and cons) and we will be bringing them up openly to discuss.

Slow roll-out of phase 2

To gain some experience with the new deployment approach and to course-correct before the “flood gates” are opened, as you write, the rollout of phase 2 starts slowly. That is also indicated in the timeline visualization.

At the risk of being redundant, I will address your questions directly:

1. How will we ensure that node operators are incentivized to be honest?

Mid-term, the IC shall be able to detect abnormal participation in the consensus algorithm. In a first step, remuneration for such nodes can be reduced and ultimately the IC might create proposals to replace such nodes. We will not have full automation for all of this ready on day 1, but we are confident to get there over time.

2. And how will we verify their honesty?

The consensus team is better equipped to give a sound answer, my try: as long as not more than a third of the nodes in a subnet act maliciously or faulty, it’s possible to identify such nodes: for example, they do not contribute to the notarization of blocks or abstain from contributing to the random beacon.

3. Is there any type of staking?

Yes, the team’s current thinking is that staking can help line up incentives.

4. Can we force secure enclave attestations of the replica, combining it with consensus, so that we can verify the replica has not been modified?

Yes, this is a project on its way and we will update this week hopefully.

5. Should we look into more automatic ways of compensating node operators? A fixed fee determined by vote seems much more subject to manipulation than a fee model that pays node operators in cycles, in addition to burning cycles.

Definitely, something we want to engage in discussion on. I will let Luis, Yvonne-Anne, Sam give some more context

[diegop]

@mparikh

Would testnet/s be also under the same purview?

Testnets are outside the scope of this, tbh. The focus is entirely mainnet. This does not preclude testnets but I wanted to be clear on the scope (“under-promise, over-deliver” and all that)

How would the concept of partial node ownership (i.e. dfinitynodes) be blended in into the node provisioning?

Good question. I am not sure if DAOs would be treated any differently than any other entities. I will let @Luis , @samuelburri or @yvonneanne answer if they saw anything different about DAOs.

[weedpatch2]

I’m hesitant to ask too much too soon on this topic, as I know it is opened up for community discussion.

I’ll start with this: I can’t help but imagine the “IC-Netboot” project that was part of the internal Dfinity Hackathon has to be tied to this automated node onboarding process. In my mind, piggybacking off what @lastmjs stated, I can see a future where any new node being onboarded (or powered on after reboot/shutdown) actually pulls the OS image from the IC on boot. That way, the OS would be verified via consensus, and all code running on the node would be validated through the IC. Also served by the IC. Also, that would make it infinitely harder to have a malicious node in a subnet, as the entire stack is served via IC.

Secondary question: With the mention of a testnet by @mparikh I got to thinking, what would be required for an IC testnet to be available to the community? Would it just require (at least 3) compatible hardware devices to be running the OS IC code and be configured to work with each other? I know that the underlying infra for the IC is pretty substantial compared to Ethereum or Bitcoin, but there are reasonable reasons to want a full testnet over just the local replica (https endpoints, PLEASE).

[yvonneanne]

How would the concept of partial node ownership (i.e. dfinitynodes) be blended in into the node provisioning?

At the moment there is no plan to tread DAOs differently from other node providers and I don’t see the need for this (happy to learn what kind of benefits there could be to do so, though). The NP rewards would go to the DAO which can then redistribute them according to its distribution mechanism.

[yvonneanne]

@weedpatch2

I can see a future where any new node being onboarded (or powered on after reboot/shutdown) actually pulls the OS image from the IC on boot. That way, the OS would be verified via consensus, and all code running on the node would be validated through the IC. Also served by the IC. Also, that would make it infinitely harder to have a malicious node in a subnet, as the entire stack is served via IC.

I share this vision for the future. In order for this work properly, progress on using nodes with TEEs (Trusted Execution Environments) will be necessary. There is a team working on this topic and there will be a motion proposal about it soon. In the meantime, we’ll work on the other parts necessary for more decentralization.

[llbrunoll]

@diegop

The original version of this post was 5x longer,

Please, @diegop , let us have the 5x version as well! (btw, thank you for your post. There is plenty of valuable information)

Also, I would like to drop a question to @lastmjs

@lastmjs

Is there any type of staking?

do you mean: to require node providers to have a minimum amount of ICP locked in their nodes to join the network?

We should have this in mind:

@diegop

Locking some amount of ICP is one of the mechanisms we’re considering to incentivize node providers, and the hardware is actually of non-negligible cost, so you can view this as staking as well.

A Gen-1 Core-Node costs ~10k $USD + extra ~20k $USD for the memory upgrade. A node provider has to be vested in ~30k $USD only for a single node! I agree that staking is already there, and it is a risk that is being taken by the provider.

In addition, a node provider will need to handle contractual risks associated with the colocation of nodes in a Data Center (usually, the deal involves at least 1 to 2 years renting contract (subject to fines in case of breach of contract).

I just wanted to say that, if we consider the required investment in hardware + contractual exposure (with the host DC) there is already considerable incentive in place for a node-provider to behave well.

Please, let me know your thoughts about that.

[lastmjs]

I agree those are good incentives. Another incentive improvement to consider is an ICP stake that can be slashed. The nodes are not specific to the IC, so I imagine could be repurposed. An ICP stake that could be slashed for bad behavior more directly ties the fate of the node operator to the fate of the IC.

[Zane]

I partially agree, but the hw and data center rent aren’t necessarily lost, you could make a deal with someone else and cover your losses or even make a profit, staked ICP can be slashed, so in the exact moment the node provider doesnt behave correctly he permanently lost money.

[mparikh]

This is also the GRT model. You do stake GRT (100k grt) in addition to node. (& you get slashed for degraded performance; wrong results in grt case). The GRT node costs are similar to IC for most performant GRT node. Of course in IC case, there’s one type of node at this point.

[namyIC]

I believe that the DfinityNodes project is on the right path to bring more decentralisation by crowdfunding nodes through their coming DAO. If all checks out on their end it would be beneficial for everyone if they got the right support and backing by Dfinity. There will always be minimum amount of hassle for anyone involved if it goes through one source and what better than a DAO that acts as a node operator backed by thousands of shareholders.

Just my 2 satoshis

[rubenhorne]

I agree that slashing could help the node operator keep skin in the game for good behavior.

My question is can we distinguish between “bad behavior” and unexpected interruptions in service (such as a power or internet outage to the node provider)? We could even imagine an organized group putting pressure on a node by threatening to cut off its power or internet and thereby lose its stake. We would want nodes to be resilient against these kinds of threats.

[llbrunoll]

@Zane @lastmjs Please, consider also (there are already → correct to → there will be) measures in place to counter bad behavior / bad performance, that is a penalty on the node rewards: penalty table .

On top of that, a single core node with storage upgrade costs ~30k $USD. Now, if we would propose to add an extra layer of investment (locked ICP associated with the node), how much would it be? 10k $USD (to make some sense)?

So, we would increase the barrier to get a new node up and running, from 30k $USD to 40k $USD.

On top of that, you can hardly get colocation in a good Data Center (with a good structure) if you want to rent a single 1U in a rack. Usually, they offer you 12 to 24U, not less than that. Meaning, a node provider needs at least 3-4U to start getting into the business.

My point is: There is already a huge financial barrier for someone to become a node provider. We are talking about ~100k $USD of the initial investment.

Do we really want to increase this barrier even more?

The community keeps saying “anyone can become a node provider” … Currently, anyone with > 100k $USD to put into hardware.

For me, it feels like locking ICP into nodes makes it even more difficult to expand the network into millions of nodes. The requirement for top-performance hardware is already a strong incentive (also, a strong barrier for new node providers); and it is already a locking as noted by @diegop .

[diegop]

Update:

Aas part of the follow-up to this post: Motion Proposals on Long Term R&D Plans (Please read this post for context).

We have begun posting Long term R&D project 1-pagers. This project will have the 1-pager in this thread.

[LightningLad91]

We should add legal fees to the cost of being a node provider since that burden is being placed on them now.

[Luis]

@llbrunoll
This is Luis Mompo Handen from DFINITY. I would like to add some slight corrections to your post.

Penalties are not yet in place. There two things that the IC needs to provide to do so: 1) automated reward payouts (roadmap coming soon) and 2) decentralized node addition (roadmap published).
With 1) penalties will be calculated by the NNS in the same way rewards are and with 2) node providers can take the full responsibilities over their nodes.

Node providers have currently three types of nodes. From about 8K$ to 30K$ whereby the 30K$ type is a minority of the network and is meant for storage subnets that are still in development. On the running costs we also have three different types depending on the number of nodes per DC and rack (6,14 and 28). I think long term there will be many types of nodes with various costs and therefore various rewards.

The deposit neuron for node providers is an idea that IMO will need more discussions in the node provider community. I would like to be careful with things that would overregulate a market for no reason. As long node providers are supporting the IC in the way they are currently doing and seeing a long list of interested node providers I don’t see a reason for such a deposit. I think we all agree that growing the network is the best we can do to ensure the resilience of the IC. Getting accepted as a node provider by the community and thereby by the NNS should be a highly valuable accreditation and of course a honor. Ideally this would always make any kind of regulation obsolete.