Subnet Management - General Discussion

No. I’m suggesting that folks would need to pay for it themselves. Any attempt to raise the capital would require legally binding terms and conditions where the contributors need to be paid back and the node machines would need to be put up as collateral.

Okay, in this case I guess I’m not sure how this relates to funding IC governance decentralisation. Maybe it doesn’t? I’m only asking to make sure I understand correctly.

Thanks @SvenF. The IC API doesn’t identify DFINITY as the owner.

  • at1 owned by ‘Flexential’
  • tp1 owned by ‘Flexential’
  • fm1 owned by ‘Hurricane Electric’
  • sj1 owned by ‘INAP’

Presumably this is due to the nature of the private agreement.

In any case, either the details held is the IC registry are misleading/wrong (though this is expected to be the source of truth) or the statement above is incorrect (how can we rule out another typo?)

I’m not clear on how any of this can be verified, either as

  • information that’s registered and reportable by the protocol, and/or
  • common knowledge that’s long been in the public domain and would have been disputed if it weren’t accurate

You can ‘verify’ the presence of something somewhere, but you cannot ‘verify’ the absence of something anywhere. Verification is an absolute thing based on proof. It’s not a probabilistic thing based on odds (unless those odds are practically zero). How many channels of communication have you monitored, and what assumptions are you making?

As far as I can tell, these proposals have been adopted without verification, which seems to be a recurring pattern. This concerns me (which is the only reason I’m mentioning it - I’m not trying to be intentionally difficult).

@Lorimer after all this clarification seems fair to adopt these proposals. The “typo” has been fixed and will be resubmitted. We all agree that a prior forum post before the proposals would have been the preferred way to do this, but at least we now have confirmation and clarity, offered by @katiep , @sat and @SvenF . I get your point that you do not trust anyone and anything so you can not in good faith vote to adopt these and probably the “upcoming proposals related to this” and that is fine. But some of us do trust Dfinity and are able to vote in good faith on proposals with well founded motivation, like in this case.

4 Likes

Thanks @ZackDS. I see where you’re coming from too. I’d like to add that I do trust DFINITY, and the excellent engineers and other members of the foundation. I trust them based on personal interactions, and I trust them based on incentive alignment. But the word trust really doesn’t belong here (or the IC is building redundant tech).

What we should be doing is applying pressure to improve processes. It will not be this way forever (a single big player that is very well incentivised to try and do the right thing). We should be recognising the cracks in the governance system and related processes so that they can be closed (sooner rather than later). Accepting the lack of verifiability and adopting things anyway is not an effective way of recording these problems and making them known and visible. Proposals are instead being adopted by the community without verification, and recorded into permanent history.

Followers follow so that this hard work is done for them, otherwise everyone would just vote ‘yes’ all the time in good faith. This is in fact a problem, and it’s not what formal reviewers should be doing (that’s what the followers could do very easily themselves).

3 Likes

We’re opening up more subnets, and plan to continue doing so in the coming weeks:

2 Likes

@Lorimer @ZackDS the fact that for these DCs a separate approach was taken I agree is confusing, and this one of the reasons why these contracts are not renewed. With these contracts ending, the IC only has Gen1 node providers and Gen2 node providers all following the same approach (submitting Node Provider proposals, DC proposals, onboarding of nodes, submitting self-declaration, etc). These contracts ending and nodes being off boarded is a significant step towards improved decentralization of the IC.

3 Likes

As I said in that review since the 48 months contracts are approaching the deadline the Node Providers are required to follow this Steps for Gen-1 Node onboarding after 48 months. The first step consists of making a forum post in this thread. If you look in the thread you will find all the forum posts that have been made for certain NPs and in which DCs they are. For the DCs that are present there is still a maximum number of nodes under the new remuneration model of 42 and the excess nodes will be removed if there isn’t a handover to another NP which will also need to be posted in this forum thread. As far as I can see the only problem in this situation was the lack of contracts to verify. No NPs have come forward and they have been cooperating looking at the channels with DFINITY which tells me that there no fault in play here. Maybe I could have reached out to some of them but I only have the matrix of the ones that have come forward in the thread. Hope this answers your questions regarding my review.

2 Likes

Sure, but unless I’m still missing something, nothing in that thread can be used to verify these proposals →


… and how did that line or reasoning work for proposal 134282? Did an NP come forward to highlight the error prior to CodeGov adopting it? If that line of reasoning wasn’t sufficient for tackling that proposal appropriately, why do you still consider it an appropriate line of reasoning elsewhere?

Hey @LaCosta @ZackDS @timk11 please do not feel obligated to engage with Alex further on this discussion on why CodeGov decided to adopt these recent Subnet Management proposals that he chose to reject. You have already provided sufficient justification for your votes. I appreciate that you asked the proper clarification questions before you voted and used the feedback provided by people who know the backstory, the lack of feedback from affected node operators, and an understanding of the work process to inform your decisions. I believe you made the right decision. Alex is entitled to use different tactics to arrive at his decision including holding extreme idealistic opinions on how proposals should be executed. This is why we have as many people making informed decisions as possible. Different people will have different opinions on what is right. I have no concerns about how you arrived at your decision and feel like they are fully justified.

1 Like

It’s been almost a week since I first expressed an interest in CodeGov acquiring the vacant node allowances that are being created by the decommissioning of data centers FM1, SJ1, and TP1 and the removal of the node machines by DFINITY. As far as I can tell, this amounts to 77 nodes total since that is how many nodes exist in those three data centers (@SvenF @katiep @sat are you able to confirm if this assumption is accurate). Nobody has stepped forward other than CodeGov to request permission to take over those node allowances. I believe proposal 132553 explicitly states that the node count is not intended to go down for Gen 1 nodes after 48 months as I referenced in this previous post. Hence, I would like CodeGov to proceed with staking a claim for up to 42 node allowances, which may be filled any time within the next 2 years. Unless DFINITY agrees a Motion is unnecessary, it seems appropriate to ask the NNS to approve this request via Motion proposal.

I will readily admit that it is unlikely that the majority of these node allowances will be filled. The biggest hurdle is that there doesn’t seem to be any servers that meet the exact Gen 1 specs available to purchase in the refurbished server market. Even if some do pop up, the remuneration reduction that will apply over the next two years would make it a challenge for the purchase of refurbished servers to be cost effective. Another challenge is that CodeGov does not currently have the CapEx required to buy this many servers, which means there would likely need to be loans involved. I like the idea of raising capital by proposing short term loans that would be paid back with healthy interest to folks in the ICP community who are willing to invest, but I don’t know if that will be successful. Hence, there are many challenges that could prevent filling these node allowances if they are granted. Nonetheless, I would still like the opportunity to try to fill these node slots over the next 2 years.

I believe this has to start with receiving permission to allocate up to 42 Gen 1 node allowances to CodeGov. I’m not certain if DFINITY can grant this permission directly (since they granted node allocations to node operators who stepped forward previously to meet the new topology), or if this is an unusual enough request that it warrants an NNS Motion proposal. If a Motion proposal is necessary, then I would appreciate feedback from DFINITY on whether or not DFINITY would vote. DFINITY triggers more than half the voting power in the NNS, so it would not be a successful Motion proposal if DFINITY has a reason to reject. I would appreciate knowing if DFINITY is opposed to this idea before a proposal is submitted. If DFINITY decides to adopt, then that is fine with me for obvious reasons. If DFINITY decides to abstain, then it would be interesting to learn how the rest of the NNS would vote. I can’t be certain how the voting would turn out, but I do think there is a reasonable chance for success since CodeGov performs a public good that helps advance decentralization of the NNS with our reviews and independent voting on technical topics. If there is no other discussion on this proposal here on the forum, then I’m inclined to submit an NNS proposal by this coming Friday. It is now December and Gen 1 contracts for the first 48 months are coming to an end. Hence, I’d like to make this request before the transition to the new contracts begin.

Hi Wenzel,

Could you explain further what this would look like?

If there were nodes belonging to CodeGov (using capital fronted by the IC community),

  • where would those nodes be located?
  • who would oversee the day to day running of those nodes?
  • what actions would CodeGov be taking to manage those nodes?
  • who in CodeGov would be responsible for that and what experience do they have managing IT hardware?

If a motion is to be submitted focusing on CodeCov, don’t you think there should be more publicity first asking for interested parties to step foreward (much like how the Grants for Voting Rewards Elections were handled).

These are great questions Alex, most of which will need to be answered at a later time after plans are firmly developed. I am already stepping forward to publicly ask for assignment of 42 nodes that will otherwise be vacated. Since I am the owner of and personally responsible for operations of CodeGov, I am empowered to make this request and to develop the business model that would result if granted these node assignments.

Proposal 132553 explicitly states that the purpose of the new remuneration model for Gen 1 nodes is to avoid the downsizing of Gen-1 NPs because it “is crucial for the community’s long-term success. Downsizing could undermine the IC’s readiness for growth, damage investor sentiment, and increase costs due to the need to onboard new NPs. Given the economic advantages of Gen-1 nodes over Gen-2 nodes, sustaining their operation is vital to avoid network capacity loss and inflation costs associated with new node provider onboarding.”. It also states “while reassignment to a “New” NP is mandatory, these nodes may continue in the same or a different data center and/or country.” This means there is no requirement for the nodes to be moved to other countries for the purpose of topology alignment. In fact, the target topology has already been reached. At this time, I think all countries are potential candidates where CodeGov nodes may be located. I will evaluate many options and the locations that make the most economic sense will be chosen.

Regarding the other questions, I would be responsible for setting up, running, and managing the nodes just like any other node operator. I will make sure the data center has remote hands available for any hands on requirements and I will make sure the proper remote monitoring tools are in place for routine off site support. While I don’t know the resources that will be available at any local data centers yet, I am aware of multiple people who I could contract with for the routine regional or global support. Racking instructions for Gen1 nodes are already available that can be followed by remote hands at any data center. I am also aware of multiple people who have the proper IT skills and are willing to help spec the equipment and travel to perform the network setup, jump host configuration, and node installation. Hence, I am confident that the required support will be available. Any final decision on data centers would include verification that contractors providing support are capable of supporting nodes in these locations.

Also, while I like the idea of seeking support from the IC community, that doesn’t mean it will actually happen. I listed many reasons in my previous message about why it may not be possible to fill all these node slots. If these node slots can be assigned to CodeGov, then I will begin fleshing out the details. Many options are on the table. If a time comes that I actually need to publicly ask the IC community to help fund this through loans, then I will present a fully baked business plan. After potential lenders are identified, we will enter into private loan contracts that satisfy the needs of all parties.

Please make no mistake about my intent. I am not doing this as volunteer work and I am not asking anyone else to perform volunteer work. I will not be pursuing it further if I can’t come up with a business model that makes sense. CodeGov provides a public good for the IC, but we do it as one entity working to advance decentralization by doing a job that pays. Everyone involved will be contracted to perform specific roles with agreed payment terms just like every other contractor who already participates with CodeGov. I know enough to think becoming a node provider is possible, but I don’t know all the details yet. My intent is for a node allocation to be assigned to CodeGov so I then have justification to develop a plan that makes financial sense. If I understand the situation correctly, there are more than 42 nodes that will be vacant. Others are welcome to seek allocations based on the same arguments that I am making for CodeGov.

@wpb I really appreciate your work and contributions as CodeGov, and I would like you to be able to continue doing the same quality of work (or even better? :rocket: ) in the future. That requires funding, absolutely. So would be good to have that solved, and I will see what can be done about that.
In the meantime, for the node allocations I was planning to propose running an ascending clock auction to use this allocation to further improve node diversity. I haven’t yet prepared a forum post for it, but I roughly plan to make it as simple as possible.
I’ll share the details here when I have some.
In the meantime, I would appreciate if you don’t submit motion proposals since that would be a direct conflict between the two ideas and it would be better to discuss the proposal here first before the submission.

3 Likes

No problem @sat. I will cancel plans to make my proposal. I didn’t know that DFINITY already has plans to allocate those nodes. I had the impression that they were going to be vacated in the name of inflation reduction.

To be honest, I really wish DFINITY would make it a prerequisite that auction bids can only be made by people or organizations that have demonstrated some sort of active participation in the protocol beyond just throwing down a bunch of money as an investor. Becoming a node provider is the only incentive offered by the NNS beyond liquid democracy voting, which takes almost zero effort, and being a node provider is currently accessible to the rich only. Allocating these nodes through an auction probably means only the rich will win yet again. It is what it is. We need whales in this ecosystem too. I just wish this auction would be scoped where others also have a chance to win.

3 Likes

Proposal 134334

A private subnet has been set to public:

3 public subnets have been set to private:

90% of the bids came from people who are already active in the ecosystem… including the winners.

2 Likes

I see there is a new set of subnet management proposals, each linked to a new forum thread rather than to existing subnet-specific threads as previously discussed.

@sat @marc0olo @cryptoschindler We never quite got an answer to this question:

Could you please respond specifically to this?

@LaCosta @ZackDS @Lorimer @wpb

2 Likes

@sat @SvenF In the course of reviewing the current subnet management proposals I’ve noticed that there are several nodes in the MN2 data centre marked as “Degraded” or “Offline”. Is there a known issue affecting this data centre?

1 Like