Cross posted from the other thread, as these queries are more appropriate here:
(i) If we had already purchased the capabilities of 1300 nodes, why didn’t we deploy all 1300 nodes? Seems a waste of tremendous money ; given that we provided generous rewards for nodes to essentially do nothing for a year. Specifically what does it mean for "the nodes to exist "? That they are simply racked in the data center, not powered on?
(ii) Now that we are onboarding the legacy 800 nodes, i am assuming that they will NOT be conforming to AMD-Milan. So when we say "[quote=“Luis, post:68, topic:9170”]
to ensure that the network is growing with nodes supporting SEV-SNP attestation.
[/quote] ", are we to presume that that growth will be in addition to 1300 nodes?
(iii)For how long are we liable for the cost of first generation nodes? These are the MOST expensive (counter to Moore’s law). Is there a plan to phase them out? Given Moore’s law, we should be replacing these nodes with next generation nodes
The first data centers were asked to have certain certifications. I personally would like to discuss with the NP community which certification could make sense and drop all prerequisites until there is a need for.
This should NOT be a question only for the NP (node providers) but also for the NNS.
I say this because the community needs to weigh in in what they would be comfortable in in context of the defi explosion that is about to hit icp-btc integration.
We expect NPs to behave within specific operational , security and performance envelopes. They in turn rely on data centers to provide some of that basis. So i think having a complete transparent idea about how we expect node providers to behave as well as provide service could be a good start.
All nodes were purchased and shipped. The onboarding on the nodes was supported by our platform ops team. The was no need to for 1300 nodes from launch on. We have nodes that are contributing, nodes that are ready to be onboarded and datacenters were nodes were racked but not deployed yet. The nodes are moving from one state to the next depending on the needs of the network.
It perhaps sounds like a lot payed but unused nodes were avoidable but reality is a bit more complex. If you already tried to roll out 1300 nodes in 70 datacenters within a year including contracting and purchasing, and that in a decentralized way, you know that this already very ambitious. On the end it took almost 18 months. Including everything that is needed to grow from there.
Correct.
Seeing the rising costs for new HW the new nodes could get more expensive than the previous models. There are no plans to replace old nodes. For sustainability reasons we would like to run them as long as possible. I think that the node rewards should allow the node providers to decide about the replacement. Rewards need to adapt to the performance of a node. Getting less rewards compared to new node types that perform better would allow node providers to decide about their break even depending on their amortisation and running costs.
The truth is that physical hardware assets are very difficult to justify to get off the books; if we are paying them a fixed price per month. I presume that we are paying the non-performing (non-active) nodes a fixed price because they are not even on-boarded yet.
Again we have rewarded the first node providers with most generous rewards IN PERPETUTITY; if there is no plan to phase them out. In that case, after the Amortization, they could be milking these nodes for the next 20 years? Is that correct assessment?
Not quite. There are no guaranteed rewards. The NNS could, at any point in time, reduce the rewards. All it takes is a proposal (and probably a discussion around it).
Rewards go down based on how much hardware can bring in to the network. If you use old hardware (gen 1) when the current architecture is let’s say gen3 already, you wont get enough rewards to justify running that old architecture (probably wont even cover electricity costs). That way, it will naturally make users either switch off their machines or upgrading to a newer architecture and better hardware.
As far as I know that’s not how it currently works, a subnet is as fast as 1/3 + 1 of the slowest nodes, so if majority of nodes are gen1, there would be no benefit in adding gen3 hardware, there are also no mechanisms to punish underperforming nodes other than kicking the node from the subnet and it has to be done manually, moving forward we might see gen1/3 only subnets so for the foreeseable future I think gen1 HW will be economically viable, the only way to eventually fase them out would be to lower rewards via NNS proposals, that is unless Dfinity adds new system to dynamically incentivate use of better HW and punish underperforming nodes.
Regarding proposal 58352 and the link in that proposal to this post, since this proposal is to further decentralization it would be nice if dashboard links to the nodes being added/removed is included in the proposal in the future.
If the general location of the nodes being added/removed are also included in the proposal body and links are provided, then its much easier for voters to verify the node changes and ensure that this isn’t just adding more US nodes and deleting nodes in places they are needed.
By the way, I manually looked up the nodes being added/removed and looks great to me!
For those who are interested it removes 1 node in each of Las Vegas, Dallas, and Orlando, and adds 2 nodes in Maribor and one node in Ljubljana (both Slovenia)
@diegop It’s worth noting here, and I am confident that this has come up in your internal discussions at DFINITY, that number of node providers is at best a necessary but not sufficient condition to achieve decentralization.
In a resilient system, the nodes are orthogonal along several vectors (eg. political risk, legal risk, electrical grid risk, military risk, and so on.)
It is a core tenet of deterministic decentralization that it is not just about the nodes, but maximizing their decentralizaton across many vectors (jurisdictions, geography, owners, etc…).
I know the node team has on their backlog on adding a decentralization index per subnet so we can see decentralization improve as nodes from different locations, entities are added
Hi all, we would let you know about a planned update to network economics
As part of the planned further automation of payouts of node provider rewards in June, we suggest tightening a network economics parameter to increase security.
In particular, we suggest changing the security parameter which controls the maximum reward to a node provider in a single distribution event from 1mn ICP to 100k ICP.
The choice for 100k ICP is motivated as follows: The maximum reward paid out to a single node provider in the first half of 2022 was 8.2k ICP in May. Thus, even if node providers add further nodes and taking into account fluctuations of the ICP/XDR exchange rate, we believe that 100k ICP should be a reasonable upper bound for the rest of 2022.
Mid-term we should consider changing the upper bound to a XDR denominated parameter as node provider rewards are paid out in XDR.
Please let me know in case of any questions or comments.
As part of the planned further automation of payouts of node provider rewards in June, we suggest tightening a network economics parameter to increase security.
How would a node provider “get past security” to receive so many rewards? Doesn’t that indicate a flaw in the node provider onboarding process, or is this just defense in depth in preparation for the highly unlikely worst case scenario?
As a node provider, I am of course sensitive to the survival of the IC and understand the desire to change the upper bound of node rewards if the price goes too low.
All that said, NPs signed up taking on a great deal of risk with the assumption that they would not be rug pulled. Just like everyone else, NPs have lost money as the market has crashed. I have not sold a single ICP token and am WAY down on my investment just like everyone else. That’s not to mention the amount of time and effort that went into setting up the racks which is substantial.
I believe it would only be fair to compensate NPs at a later date if caps are implemented today and token rewards are missed out on due to a declining token price. We all are sensitive to the prospect of a death spiral which is in no one’s interest. That said, if a cap is implemented the additional inflation should be awarded at a later date, perhaps over time in neurons etc.
With seed investors, the DF didn’t say, woah, the price has gone up too much, we can’t give seed investors all those tokens they purchased. Why? Because that’s the deal they signed up to.
Doing this is not only the right thing to do. It also makes sense from a security standpoint. It will be incredibly harmful to the security of the IC if NPs do feel the community is ready to rug pull them with no compensation at all. It’s important that NPs keep their machines on and future NPs are excited and confident their investment will be honored by the community.
I would like to clarify that the proposed cap adjustment is of pure technical nature in the context of the automation of the payouts of the node provider rewards. It is a defence mechanism against potentially erroneous behaviour of the protocol.
The proposal is not a change to the tokenomics of the IC (or a protection against inflation). This would be (as you already stated) a separate discussion.
It’s about time for the next update on our platform decentralization roadmap. A few weeks ago we published the node deployment runbooks for the Dell and Supermicro nodes. In the upcoming 8-12 weeks the node providers will start redeploying their nodes with this process. You will see many node replacement proposals freeing data centers in preparation of this redeployment. Per data center we need 14 or 28 (depending on the data center size) proposals to add replacement nodes to the corresponding subnets, another 14 or 28 to remove the nodes from the corresponding subnets and one last proposal to remove all nodes of a DC from the inventory. That’s 29 or 57 proposals per data center that is going to be redeployed.
Quick reminder, since launch the node deployment procedure was as follows: (1) DFINITY pushed the latest binary to the node machines, (2) the node provider locked down the node, i.e. updates are only possible through the NNS, (3) the node provider initiates the key material generation. At launch this was the only deployment method available. In order to increase the platform decentralization node providers can now set up their nodes completely independently.
We are using this redeployment to also complete the storage upgrade of some nodes. Some data centers already have type 1 nodes, others still run type 0 nodes that need the storage upgrade from 3.2 TB to 32TB in order to become type 1 nodes.
First EU NP will be @awrelll 's data center in Bucharest (BU1). First US datacenter will be in Miami (MM1). First Asian data center will be in Singapore (SG1). The order afterwards is not determined yet and depends on how quickly the previous badge managed their redeployment.
What’s next?
As I promised in my previous status update we are almost done with the specification of the next node type. I’m expecting to get the specification published by the end of this week. The new node specification is now generic. The first three node types at launch were specified based on BOMs of Dell and Supermicro. This was necessary to avoid too much diversity on a platform that we first need to prove that it runs stable. We proved that and specifying a specific server model from a specific manufacturer adds a centralization that we now can remove.
The decentralized data center setup is also ready. This basically means that we will publish a node connectivity specification until the end of the week for new node providers and a migration runbook for existing node providers some weeks later. In addition we started working on a best practice white paper to allow node providers to access their nodes remotely and securely. Once the node providers rolled out that remote access we reached the milestone to implement decentralized SRE procedures. Stay tuned for more details on my next status update.