SYBILing nodes! šŸ˜± Exploiting IC Network... Community Attention Required!

Governance NNS Governance
, ,
85

Proposals 135664, 135665 & 135666 | Tim - CodeGov

Vote: Reject

These proposals are intended to replace some of the nodes in 3 selected subnets. I appreciate what the @DRE-Team are trying to do here in providing a option to split up allegedly colluding node providers so as to reduce the chance of a successful attack on one of these subnets. I also think the ā€œclusterā€ approach is worth considering. However, Iā€™d first like to see more community discussion on this approach and an NNS vote on whether this is the way that weā€™d like to move forward. While the uncertainty remains, Iā€™d also prefer to keep the Nakamoto coefficients high. The Nakamoto coefficient is the minimum number of entities (such as countries, node providers or data centres) who would need to collude in order to launch a successful attack. In each of these proposals the regional and country Nakamoto coefficients decrease, and the number of countries containing 3 nodes for each subnet increases, thereby leaving the network more susceptible to attack than before.

All that being said, however, I wonā€™t argue against anybody elseā€™s decision to adopt or reject these proposals. Although Iā€™ve voted to reject them, Iā€™m glad to see these proposals submitted. The big thing about this is that it provides the NNS community with a option to change the network or not to change it based on which perceived threat is thought to be the greatest, and to do it in a way that doesnā€™t render the network unstable. What Iā€™d most like to see here is for a high proportion of voters to weigh this up and cast a manual vote rather than leaving it to the major followee neurons to make the choice.

About CodeGov

CodeGov has a team of developers who review and vote independently on the following proposal topics: IC-OS Version Election, Protocol Canister Management, Subnet Management, Node Admin, and Participant Management. The CodeGov NNS known neuron is configured to follow our reviewers on these technical topics. We also have a group of Followees who vote independently on the Governance and the SNS & Neuronsā€™ Fund topics. We strive to be a credible and reliable Followee option that votes on every proposal and every proposal topic in the NNS. We also support decentralisation of SNS projects such as WaterNeuron, KongSwap, and Alice with a known neuron and credible Followees.

Learn more about CodeGov and its mission at codegov.org.

5 Likes
86

DFINITY Foundation appreciates CodeGovā€™s feedback on the submitted node swapping proposals and its commitment to ICP network resilience. We understand and share your concerns about preserving higher decentralization.

At the same time, we believe that moving forward with these proposals is the best step precisely because it allows more community input on the matter while removing (however small!) any potential risk to the ICP in the short term simply as a precaution, as articulated in this thread.

We are willing to work with the community to create a long-term solution to the problems we currently face, including a fully transparent and algorithmic way of calculating the best network topology strategy and effectively capturing connections between various node providers to strengthen the decentralization of the Internet Computer.

1 Like
87

No problem @alexu. @timk11 is one of our reviewers out of 3. I asked all of them to vote with their own convictions on these proposals no matter what it is and regardless of the DFINITY position. I believe one of our other reviewers will vote to adopt for specific reasons, but he will announce that when he is ready. I donā€™t know the intentions of our other reviewer yet, so I donā€™t know how the CodeGov neuron will vote. We will vote according to the consensus of our reviewers though. They each have solid reasoning in my opinion.

Personally, I was surprised that DFINITY decided to remove these node providers from these subnets at this time given the lack of evidence of foul play. However, I understand the decision to err on the side of caution. Itā€™s unfortunate that it gives a sense of credibility to claims that fell significantly short of what should be expected by anyone looking at it through an objective lens. It makes the node providers look guilty and there is no justification for it.

3 Likes
88

I have a feeling that only a small proportion of our 30 node providers are actually guilty.

2 Likes
89

Fully agree with your reasoning and decision.

2 Likes
90

Can you explain the very early vote that is actuall an F u to community, reviewers and all that Decentralization of the IC was supposed to stand for ? Thank you .

1 Like
91

It aint decentralised. Im fixing that.

3 Likes
92

Itā€™s unusual for Dfinity to vote so early, and in this case Iā€™m surprised given the opportunity that was (apparently) being put forward to the community to take the evidence on board and make a choice on this. As explained in my review, I was hoping to see which way this was leaning before Dfinity went ahead and fired off all the follower votes. This does not look like decentralisation in action.
@sat @alexu @katiep @dominicwilliams

2 Likes
93

Thanks @Sat

Iā€™ve voted to adopt 2 and reject 1 (one which replaces a healthy node with an offline node). As noted by others, the proposals have already executed.

For those in the community that are interested in detailed reviews, Iā€™ve provided links below. The rest of the CO.DELTA team will be posting their reviews and voting shortly.

For those in the community that are looking to follow a decentralised neuron thatā€™s also backed by a team that put the work in to review these sorts of technical proposals, consider following CO.DELTA ā–³. Weā€™re almost finished with testing and finalising the controller relationship between the canisters to enforce decentralisation.

94

Thanks @geeta23, I noted that @paul23 (who onboarded in 2024) decided to follow the convention set by you and @tina23 (and is also linked to this group in other ways). However at this stage itā€™s nit-picking on my part.

Thanks @GAbassad, I think this would have been whatā€™s necessary to uphold the node provider ā€˜independenceā€™ expectation thatā€™s assumed by the IC Target Topology. Iā€™m still not clear on why a distinct business entity needs to map onto a distinct Node Provider entity. However, I guess onboarding half of those nodes under your personal NP and half under @tina23ā€™s would still have been undesirable in other aspects (it wouldnā€™t fully capture the reality of who has control over those nodes).

I really appreciate your responses. I think the NP-links suggestion that @sat came up with sounds like a promising way of modelling this sort of thing moving forward. @Lerak seems to be doing a great job of pushing this idea forward :+1:

Iā€™d like to try and turn attention to a question that has been asked repetitively, but still an answer has not been provided.

  1. Why are some node providers feeling the need to obscure the terms of their node transfers to other Node Providers by getting them to sign a non-disclosure agreement about the terms of the node transfer?
  2. How can anyone expect the original NP to be fully relinquishing their claim to those nodes under this sort of circumstance?

Can the community agree that we should be requiring the details of how an NP acquired their nodes? @SvenF seemed to be of this opinion before he left DFINITY. I think this should be considered a high priority item, and node-onboarding-reviewers should start requiring this information. I think it should also be demanded retrospectively for some of the recent onboarded NPs. Does this sound reasonable, or not?

In my opinion it doesnā€™t matter that this information could be spoofed (it can still be used as evidence in the future, as a means of holding NPs to account in the course of physical and/or paper audits).

Iā€™d also like to resurface the topic of automatic node shuffling and bring it to peoples attention (as a means of helping to secure the IC as it scales). More critical thought and debate are needed.

3 Likes
95

I think this information needs to be public, so I am posting it here. Adam, who goes by @borovan here on the forum, and was the proposer for proposal 135636, asked the basic question, ā€œhow do you find out who voted no on 135664?ā€ and commented that he hopes that I am not part of the group. I am part of the group as described below. Sensing that my response may not be what Adam wants to hear, I want to make sure that this answer is very transparent to the community. I appreciate having a DM with Adam, but I also know that he can jump to conclusions and sensationalize reality with conspiracy theories. Hence, I consider this to be a necessary post to pre-emptively address any concerns that he may raise.

The short answer to the original question is that the votes of all known neurons can be found at the very bottom of each proposal on the dashboard.

My involvement is as follows:

Synapse and CodeGov are both neurons that I started. I organized a group of people to form Synapse 3.5 years ago at the time when known neurons were first allowed. We were the very first community based known neuron. It was called ICP Maximalist Network after the chat group where we all met and discussed ICP extensively. It was later renamed Synapse in order to separate the known neuron identity from the person who calls himself ICP Maximalist. He wanted to dissociate his brand from ICP politics because he was in the process of starting up what is now called BoomDAO. He hasnā€™t been involved in Synapse for over 2.5 years now. Regardless, I was the primary lead for the Synapse neuron and still help keep the voting members of the neuron organized and motivated to participate in governance.

About 2 years ago I realized that volunteer participation in governance isnā€™t going to be effective and sustainable long term and wanted to start looking for way to motivate developers to get involved in governance on technical proposal topics. Hence, at the advice of legal council and CPA, I formed CodeGov, LLC. DFINITY provided grants and I recruited developers to review and vote independently on what is now called IC-OS Version Election proposals as well as what was previously called System Canister Management proposals. When the Grants for Voting Neurons program started, we expanded into Protocol Canister Management, Subnet Management, Node Admin, and Participant Management proposal topics.

Our goal is to advance decentralization of the internet computer by reviewing these technical proposals and voting independently in a credible and reliable way on every proposal submitted to the NNS. The CodeGov NNS known neuron has recently added voting members for the Governance and the SNS & Neuronā€™s Fund proposal topics as well, but they are all volunteers since there is no funding to review those proposal topics. CodeGov has also branched out into SNS governance participation, again as volunteers since there is no funding, and we have established known neurons for the WaterNeuron SNS, the KongSwap SNS, and the Alice SNS.

In all cases, the primary objective of CodeGov neurons is to advance decentralization by putting people with the right skill set in a position to actively cast credible, informed, and reliable votes for every proposal that we support. In the event that we cannot reach consensus by the last day of the voting period, then a manual vote is cast based on the majority of votes cast by our Followees to that point. My role is to actively manage these neurons and voting members to ensure we have diplomatic and skilled resources in place to review and vote on proposals and to seek funding to incentivize this work as much as possible. It is real work to perform these reviews and to manage an organization and I believe that true decentralization will not occur unless the proper incentives exist. CodeGov intends to be one of many neurons who votes on governance proposals. We support and advocate for many known neurons to be incentivized to perform the work of credible and reliable decentralized decision making for NNS and SNS projects.

Hence, yes, I am involved in some of these neurons that have voted to reject proposal 135664. I stand behind the decisions that our reviewers have made on these proposals. They provided their justification on the forum. There is no proof of collusion among these node providers and the evidence that has been presented is very weak and easily explainable. Perhaps we just need to agree to disagree. I do understand your perspective since you are a very large whale in this ecosystem. It makes sense that you want to err on the side of caution and remove node operators that you have accused of collusion. However, I believe that myself and all of our reviewers at CodeGov are able to look at these accusations through a much more objective lens and we are not seeing any justification for these changes based on evidence presented. Our job is to provide an independent voice and that is what we have done.

The CodeGov known neuron reached consensus and voted to Reject proposals 135664, 135665, and 135666. Our final tally is 0 YES and 3 NO on all 3 proposals. The CodeGov vote was triggered by our reviewers (@timk11 @ZackDS @LaCosta) for the Subnet Management proposal topic and their reviews are posted above in this thread.

The Synapse known neuron reached consensus and voted to Reject all three proposals as well. The final tally of our Followees was 1 YES and 1 NO on all 3 proposals. The Followees for the Synapse known neuron on the Subnet Management proposal topic at this time are CodeGov and CO.DELTA (the neuron started recently by Alex Lorimer). The 3 voting members for CO.DELTA (@Lorimer @aligatorr89 @MalithHatananchchige) have also posted their reviews in this thread as well.

In all cases, these known neurons made valid decisions that were cast by educated individuals who are aware of the issues and voted according to their own convictions based on the evidence. I stand by the decisions made by every person involved.

6 Likes
96

Requiring the accuser to provide direct evidence of collusion between node providers is unreasonable thing to ask for. If a group of individuals coordinating via Telegramā€”without any official connectionā€”executes an attack, no such direct evidence would exist except one that looks like a conspiracy theory. The key question should be whether the NP selection process could have been gamed. If it could have been, that alone justifies taking precautionary measures.

At first glance, it appears that if attackers control 1/3 of the nodes in the SNS subnet, they could submit treasury withdrawal proposals and then deny all blocks where someone attempts to vote against them. The NNS could intervene and remove these malicious nodes once they reveal themselves, but if the attackers also control 1/3 of the nodes in NNS subnets, then proposals to remove them wouldnā€™t pass either.
(Edit: Attackers will also need SNS tokens since treasury proposals are critical - ā€œAt the end of the voting period, a critical proposal is adopted if more than 67% of the votes cast are Yes votes, provided these votes represent at least 20% of the total voting powerā€)

Secure enclave nodes will make such attacks significantly more difficult. Before that happens the community has to remain vigilant.

3 Likes
97

Proposals 135664, 135665 & 135666 ā€“ LaCosta | CodeGov

Vote: REJECT

There has been no evidence for colluding nodes other than speculation. I donā€™t think decentralization should be ruled by conspiracy and as I stated in a previous review

I think this has been proved correct. Even with clarifications from NPs such as pictures, videos and bills, there are some people in the community that donā€™t seem satisfied and I doubt they will ever be.
I think the best solution lies in third party audits done by a company voted by the community. This will be periodical and will stop this marketing campaigns targeted against NPs based only on speculation and username choices.

About CodeGov

CodeGov has a team of developers who review and vote independently on the following proposal topics: IC-OS Version Election, Protocol Canister Management, Subnet Management, Node Admin, and Participant Management. The CodeGov NNS known neuron is configured to follow our reviewers on these technical topics. We also have a group of Followees who vote independently on the Governance and the SNS & Neuronā€™s Fund topics. We strive to be a credible and reliable Followee option that votes on every proposal and every proposal topic in the NNS. We also support decentralization of SNS projects such as WaterNeuron, KongSwap, and Alice with a known neuron and credible Followees.

Learn more about CodeGov and its mission at codegov.org.

1 Like
98

Sure, this is possible, but there has been no credible evidence presented that this has actually happened. The only reason that this thread that Alex started has any value is because it has engaged more people in the discussion of solutions that minimize the potential problem.

The only precautionary measure that would make sense is to replace all nodes with DFINITY controlled nodes (another bad idea). Otherwise, how do you decide who is a bad actor? It shouldnā€™t be based on weak evidence. In this case, it was based on forum usernames that have the number 23 in them, or new forum usernames that were created before they made a node provider onboarding post, or node providers that used the boiler plate onboarding form that DFINITY asked them to use and the didnā€™t take the time to customize it, or node providers who copied an example of a successful NNS proposal from another node provider as suggested by DFINITY and didnā€™t customize it enough, etc. Tomorrow claims of collusion could be made based on people whose favorite color is red or people whose favorite color is blue or something equally ridiculous. Weak evidence should not be used to decide who is a bad actor.

The right answer is to back away from all these accusations and supposed evidence and start focusing on an improved node provider selection and verification process. Many people have called for this and yet we keep taking actions that implicitly judge specific node providers as guilty with no evidence. We need to stop focusing on the accusations and start focusing on improving the work process.

4 Likes
99

I too would prefer that we focus on being constructive, and fact finding / information gathering by asking questions (and cross-referencing things where necessary). This is what Iā€™ve been doing, and I hope my intentions have been clear. I am however glad that the community has started taking this general problem more seriously as a result of the proposals that were raised (not by me). Iā€™m confident that solutions are in sight if we can keep this momentum going.

The immediate questions that still need an answer (which shouldnā€™t be difficult) are:

2 Likes
100

Hi @katiep, it sounds like you may be able to add context to the questions above based on your comment on the working group thread.

What are these NDAs about preventing, and why would it make sense that a new node provider cannot disclose the details of the node transfer agreement (e.g. what they gave in exchange for the nodes, whether or not they were gifted, purchased, leased, or other).

Thanks

101

You highlighted a quote from @katiep that was referencing NDAs with data centers, which is different than the question you are asking about node provider NDAs. In regards to data center NDAs, they are likely protecting pricing agreements. Every user in the data center has to sign expensive contracts that are negotiated individually and there can be pretty big price differences based on scope. A smaller user would likely need to pay higher prices than larger users. Data centers are also big business and each one is competing for customers. They each probably have their own intellectual property and know how that they want to protect. The NDAs could range from the protecting technology behind their power and cooling distribution systems, services offered, arrangements they have with ISPs, ensuring customer confidentiality, etc.

In regards to node provider NDAs, Iā€™m not sure we can arrive at a productive result with this line of questioning. It doesnā€™t really matter why they entered into non-disclosure agreements. The parties involved felt that their business interested needed protecting, so they did. There was nothing that prevented it as far as I know.

Regardless, having an NDA in place does make it difficult to verify independence. Hence, it seems more constructive to focus on ways to improve the work process such that revealing this information cannot be protected by these types of NDAs in the future. For example, we could establish that NDAs are acceptable, but they must be declared and they must be presented for viewing during an audit. That way node providers could still protect what they feel is relevant to protect while still enabling auditors or reviewers an opportunity to verify independence. It might even be necessary to require that node providers file legal documentation with their local jurisdictions, or on the NNS, that lists specific relationships and/or NDAs. I donā€™t know what that might look like, but at least it would be a paper trail that could be used as evidence of agreements if needed later.

Iā€™m not sure that these kinds of changes could be implemented now though. The time for establishing the ground rules for what constitutes acceptable relationships and acceptable proof of independence was back in late 2023 and early 2024 before a lot of node providers had to make changes. They all made business decisions and investments based on the accepted criteria for transferring nodes and/or onboarding new nodes. It may not hold up in court if we try to change the rules now if any of the node providers are unwilling to comply with new rules and they feel they have a grievance.

It would be nice if the NNS can agree to some new criteria that could be applied now, but if not then for sure these things could apply to the next round of node provider onboarding. Perhaps the NNS could offer a new remuneration incentive in the near term for voluntarily disclosing certain information or going through a more stringent onboarding policy again. The promise of higher remuneration might convince folks to do it without changing the rules of the existing onboarding policy. Or maybe there could be a progressive remuneration penalty that is applied that starts at zero penalty and slowly increases over a couple of years.

Anyway, understanding the reason that each node provider has felt the need to protect their business or investment with an NDA doesnā€™t seem like something that will result in a productive outcome. I think identifying the rules and work process that can ensure a higher degree of transparency in the future is the place to focus. In the meantime, establishing relationship clusters is the best idea I have heard so far on how to mitigate the risk of assigning too many nodes from node providers with a relationship to the same subnet.

These clusters could be defined by node transfer parties and assumed relationships, but with no remuneration penalty. The DRE tool could take these relationship clusters into account in the topology targets. However, the NNS could offer higher remuneration to node providers who are willing to prove a higher degree of independence and can be convincing enough to be separated out of the cluster that they have been assigned.

102

I disagree completely. There should be nothing to protect in this respect if the nodes have truely and cleanly changed hands (which is a requirement). You donā€™t know the answer, so lets get one. @katiep, do you know any more about this sort of thing?

103

You are heading down the path of more conspiracy theory, which you started initially and Adam perpetuated and amplified. There is no productive end to this line of questioning as you have experienced throughout this entire thread. Letā€™s focus on being productive.

4 Likes
104

Letā€™s focus on draining the swamp.

Iā€™ve just found 35 node providers linked to the same account, thatā€™s just the start. I believe > 100 of the current node providers are part of the same group of people.