The State and Direction of Decentralization & Nodes on the Internet Computer

Hi Bjoern,

If this cap is just a technical measure in case automation fails, what sort of protections do you have in case the algorithmic stablecoin nature of ICP/cycles fails and leads to a death spiral?

This setup unfortunately reminds me a little too much of Terra/UST, although in this case an “attack” is much more difficult because users cannot simply burn cycles for ICP. However, it is not difficult to imagine a scenario where developers burn a lot of ICP for cycles, the exchange rate of ICP/XDR falls, and ICP holders are severely diluted when paying node providers.

This scenario is even more dire when you consider that we are also using cycles as a stablecoin for Defi on the IC (Dank’s XTC). If a lot of XTC is minted when ICP prices are high (you can imagine that this might potentially be orders of magnitude greater than the cycles used for powering canisters), then when ICP prices are low we would have a huge supply of cycles “backed” by a much smaller amount of ICP. And relief of this supply pressure only happens very slowly as canisters consume cycles.

Has this been a concern at DFINITY? I can imagine there are a few potential fixes that might alleviate this up to a certain extent, but I think this is an intrinsic problem with an algorithmic stablecoin that is not 100% backed. I know you want to keep the deflationary nature of ICP when minting cycles, but you cannot simply create a “stable” currency out of thin air by burning your own token.

For example, instead of burning ICP when minting cycles, the ICA could potentially sell this ICP on the market and hedge by buying XDR (in fiat currency) and keeping these as reserves. Or we could introduce some inflation for all cycles to disincentivise holding cycles/XTC.

Or the simplest solution: why not get rid of cycles altogether? Just pay for computation at a variable rate determined by the NNS? Having a stablecoin is a great idea, but I think the current implementation is a little bit risky. And frankly it is yet another project that DFINITY is undertaking that is not fundamentally necessary for the IC to function. You could use ICP, use another established stablecoin like USDC, use BTC, or even use a combination of all of these.

3 Likes

@Luis How are we progressing on the original roadmap that opened this forum thread? I can’t seem to piece together how we stand. Are we behind? Are we hitting the quarterly milestones? I’d love some updates and would love to know how to follow progress.

5 Likes

I noticed Dom tweeted about a medium post coming out imminently about node decentralization which now appears deleted ! Dum dum dum

5 Likes

@dfisher I’m not following all of Dom’s tweets and I’m not aware that there is any change in the platform decentralisation decided. I’d recommend to attend the public, global RnD. This is where such decisions would be announced. Also internally.

@lastmjs We already missed quarterly milestones end of the first quarter. I must admit that I didn’t thought about updating the complete roadmap to that and other delays.

TLDR; The delay in the roadmap is mainly caused a higher prioritisation of the BTC and SNS feature and by the decision to finish the storage upgrade on all nodes until end of the year. This also led to the decision to change the way we are setting milestones and thereby untangle dependencies in deliverables. See @samuelburri’s explanation in last public global RnD.
The total delay is mainly impacting the workflow optimisation for NP onboarding and the network growth with these new NP and a Milan-based new node type. There’s still a chance for completing that roadmap by EOY.

I’m referring to this pseudo-gantt from last year November.

Deliverable (A) Reproducible builds: I don’t have the exact time when this was delivered but the corresponding build code is public since January when the corresponding builds passed the acceptance tests. We published a recording of that here.

Deliverable (B) Autonomous Node Deployments: Technically it was done in January with the reproducible builds. The remaining work that was shown for the first quarter was mainly targeting NNS frontend DApp integrations. This was descoped in March because the corresponding team was still working on reworking the complete frontend code to svelt.
We had to rework some details regarding deterministic IPv6 assignments and regarding a VSOCK implementation that allows the replica to do changes on the host system. Due to that delay the rollout “Swap D0 nodes by D1 nodes” started end of Q2 and not end of Q1.
This also delayed the future NP onboarding which is meant to complement the current CLI based onboarding with UI supported workflow. In addition we decided that we would prefer to onboard new Milan-based hardware and not modify the existing Rome-based hardware. I expect the corresponding motion proposal to be placed soon which bring us back on schedule for deliverable (E) Next Node Hardware Generation.

Deliverable (D) NNS-driven data center allocation is still in the design phase. There two things blocking us from starting the implementation: First the NNS team is still focused on the BTC and SNS feature and secondly we found out that the decentralisation code that we currently use to create subnets and replace nodes needs a better testing (more edge cases) before we can integrate it into the NNS. The current node replacements for the storage upgrades help a lot in this regards.

5 Likes

Thanks @Luis. Also including @diegop, and flagging both @lastmjs and @bob11 given they have recently flagged the centralization of node providers as a key concern.

I think it would be helpful to more clearly and regularly update the community on how steps (A) through (D) are going, and specifically timelines expected.

Without getting into the weeds on steps (A) through (D), the community should also have a good sense of how many node providers / owners we can expect in Q3, Q4, Q1’23, Q2’23 etc. The expectation on node provider count right now is super confusing. Realize there are many components to decentralization in a node provider context that comprise steps (A) to (D) but from a basic security standpoint the number of nodes and the number of node operators is a basic one.

Mind figuring out a workflow to keep us updated and informed on this front? I do believe there is general confusion about what it is to be expected and what will happen in the coming months.

There are other threads that dive into the weeds on which hardware spec to choose and I feel like most people don’t see the bigger zoomed out picture right now.

4 Likes

And if we could get some analysis on node shuffling/rotation and get that officially into the roadmap if the benefits are clear, that would be great too.

5 Likes

I see in the subpost it is mentioned that there is a way to put a new node to the network

Can someone clearly clarify this: -

  1. Is it still possible to become a NP and provide node to network and earn rewards? without steps A to D (mentioned in this thread ) being completed ?
  2. If not then is everybody waiting for A to D to be completed, and this form Discover Typeform, where forms = fun to be updated with link to be pointed with new instructions before any new node is added to the network?
7 Likes

It is (and will always be) possible to become a new NP, but there are a few practical obstacles at the moment. One is the hardware spec; as stated by @Luis above we are working on a new (updated and less restrictive) hardware spec; the old hardware spec has been very restrictive and it may not be straightforward to get new hardware satisfying it. (I’ll try to come back with more details on the new spec.) Another (potential) obstacle is that the onboarding process is quite technical at the moment and the tooling needs improvement.

The – technically – essential parts of the deliverables (reproducible builds, software distribution for deployment) have been completed. Practically, another dependency is then the new node hardware spec.

6 Likes

If we want IC to be the L1 of the Internet, you must work hard on it and facilitate access, we need a good balance between descentr & safe of the chain

2 Likes

Okay I see your point.
Now since we agree that any one following the wiki can spin up the node. I have few questions based on few assumptions
Assumption:

  1. Lot of Votes are follower votes, so I understand if Dfinity foundation / ICA/ (whoever has majority of following ) votes then the proposal is expected.
  2. Hosting a node requires few proposals, new NP proposal, New DC proposal, and few more.

Question:

  1. If anyone submits the (“Register a New NP proposal” and other proposals) to add a node to network, on what grounds Dfinity foundation / ICA/ (whoever has majority of following ) approves or rejects it ?
  2. I see a lot of nodes are sitting in Awaiting subnet status. And as per my understanding one does not get rewarded unless the node is in a subnet. What is the criteria to get into a subnet ( either old or a new one) ? . Are subnets created automatically or manually for now?

The whole point of asking these questions is though we wait for this thread to complete there are people who want to invest and become node provider, and it comes with a hefty cost.

Request: Whoever answers this post, please try to objectively answer the bullet points and all the questions if possible. I am sure me and community will have followup questions on this.

5 Likes

Yes, personally I would like to see this happen as well. I’m willing to invest in being an NP if it becomes available for individuals to host at their residences. This was a major hope of mine for the future. I really like the idea of investing in becoming an NP if the rewards are substantial enough to make it worth while. I also would like to help decentralization in any way I can.

Please keep us (the community) updated!

4 Likes

The main purpose of having the registration go through the proposal flow is the validation of the information. That is, to add a new NP, the NNS needs to validate the identity of the node provider. The documentation necessary may differ from case to case. As this is the first time we’ll be adding entirely new node providers post-launch, this probably also deserves some discussion on the forum.

While your assumption 1. is correct, which is that at this moment DFINITY and ICA direct the largest chunk of voting power on the relevant topics, we will try our best to collaborate with the community on the decision.

Here we need to differentiate between (a) the current implementation and (b) the intended future mechanism. At this moment, the rewards that are paid out are determined exclusively by the “rewardable nodes” setting in the IC’s registry. That is, the information whether or not the nodes are in a subnet is not yet used for the reward computation; the “rewardable nodes” data structure is the source of truth. That data structure is changed via a proposal to the governance system, which is the last step in the NP onboarding flow.

At the moment, subnets are created and nodes are added/removed from subnets based on proposals, most of which are today generated by DFINITY’s team. Behind the scenes, the team uses tooling that maximizes decentralization based on objective criteria. In the future, we expect to move that code into the NNS so that all decisions on node assignment are performed programmatically by the governance system.

In the future, the remuneration will take more data such as actual subnet membership and node uptime/downtime into account, so the accounting will be a lot more precise.

6 Likes

Dear community, this is Sven Fischer from DFINITY, part of the Release Management team.

As we are moving into the fourth quarter we would like to give an update on the status of the Node Decentralization activities. To recap, the onboarding of new Node Providers (NPs) was halted due to reprioritization of other activities, amongst others, the Bitcoin Integration. However, other node decentralization activities, like upgrading of existing nodes, and testing of new hardware specifications, have progressed in the meantime.

As of the start of Q4, the status of our decentralization efforts is as follows:

  • The upgrading of existing nodes to is in progress and continues until the end of 2022. Storage upgrade will be included in this to accommodate potential growth in storage requirements of the IC network.
  • In Q4, we will start onboarding the first new Node Providers. Focus will be on adding nodes of new NPs to the NNS subnet in order to extend our degree of decentralization of the NNS. Furthermore, part of the new nodes will be used to increase the security of the ECDSA subnet used for Bitcoin integration.
  • The new NPs will be onboarded using the new hardware specification (Gen2 replica node hardware specification), while we are completing the final tests of the hardware in parallel, specifically on the support of SEV-SNP
  • We will start in Q4 with onboarding of 6 new NPs.
  • A subset of these new NPs will be asked to onboard independently based on the currently available NP onboarding runbooks (Node Provider Onboarding), in order to show the community that NPs can onboard without intervention or support from the foundation. Technically it is already possible for a NP to onboard, but there are technical practicalities as stated in the above thread that inhibit this, and we want to make the onboarding process a smooth and efficient process.
  • Self-declaration: While the new onboarding process provides more independence to NPs, it brings the question how the community can judge which NP proposals to adopt and which ones to reject. The team is preparing a self-declaration template. Through an upcoming NNS motion proposal, we propose to establish this self-declaration as a means for NPs to provide more information to the NNS community, enabling an informed vote on whether to adopt a new NP.
  • For the new NPs being onboarded we will follow the existing remuneration schema. That is, rewards are determined exclusively by the “rewardable nodes” setting in the IC’s registry. This might change in the future as the team is looking into taking into account more precise metrics such as node uptime and downtime (see also the previous thread response on Remuneration).

The DFINITY Foundation is currently working together with the community and existing NPs for the above activities. There is a large backlog of NP requests so we expect the decentralization to accelerate over the first two quarters of the next year, with the introduction of runbooks for independent onboarding by new NPs.

Q4 2022

  • (A) Continued work on upgrading existing nodes.
  • (B) Onboarding of 6 new NPs, with a subset following runbooks for autonomous node deployment. Deployment will be based on the next generation HW configuration that is currently in test.
  • (C) Testing of next generation HW for different HW vendors.

Q1 2023

  • (C) Testing of next generation HW for different HW vendors.
  • (D) Start of autonomous node deployment for new NPs.

Q2/Q3 2023

  • (E) NNS-driven DC allocation: The NNS-controlled DC allocation mechanism will be introduced and control all future node additions. With this step and a new node hardware an accelerated network growth is unleashed.
  • (F) NP onboarding using NNS Front-end dapp instead of CLI tooling.

Of course, based on community feedback or technical obstacles that we might encounter, the plan might be adjusted. We will continue to update you through the forum on the status of these activities.

16 Likes

How are those selected? Is it on a first come first served base from the backlog? If so, is there a way to find out what place one has on the waitlist?

4 Likes

We seem to have about ~74 existing NP’s, have the current NP been “fully on boarded” with most of their nodes being operational? Looking at the IC Dashboard there seems to be a lot of gaps. Do you know what could be holding that up?

Am I also correct in understanding that new NP’s will have same rewards as the first round? Although NP rewards are a smaller factor in overall inflation rate, that seems less than ideal.

1 Like

Thanks @cryptoschindler, this is a very good question. The team is still discussing on this, but it will most likely take into account different aspects, as we need to take into account decentralization metrics per city, country, region and continent; so it will not only be on first come, first serve basis. As soon as I have more information, I will make sure to update the community in this thread.

2 Likes

Real question is how will they be selected once NNS is responsible for node onboarding

4 Likes

That is correct @Zane and we will definitely come back on this.

1 Like

All this new ob-boarding process looks okay. But to be frank, when does one buy the hardware? Buying the hardware takes a lead time of up to 6 months.
So my questions are:
1.
Whoever wants to become an NP, shall they apply first to become NP? Reach step IX Node Provider Onboarding - Internet Computer Wiki
Because till then it’s all a manual approval process and no one would want to buy such expensive hardware without knowing with 100% certainty what it will take for them to reach that step.
2. Also I understand a Node is not rewarded unless it is in a subnet. Even when a node is onboarded how does the process looks like to get into a subnet? No one would want to have a node onboarded and just pay bills without getting rewarded.

Let’s try to bring more clarity to the community on this.

11 Likes

Thanks @ritvick for your questions. These are actually very relevant, and the reason for not responding earlier is that we are discussing these questions internally. We are working on clear guidance and proposals in this area that we will share with the community shortly. Thanks again for bringing this up.

3 Likes