SYBILing nodes! 😱 Exploiting IC Network... Community Attention Required!

[Sormarler]

@borovan Could you share the evidence supporting this claim? The suggestion that 100 node providers are controlled by the same group or individual represents a critical security concern. If accurate, this would indicate a significant compromise of the Internet Computer’s infrastructure. Without supporting evidence, this may be perceived as unsubstantiated speculation that unnecessarily destabilizes confidence in the network.

[marcio]

If we assume for a moment that there are clusters of node providers belonging to the same or related entities, what can you legally do about it? If I read correctly, they were allowed to have at least one node under their identity and another under a company name. So even if you tell them, ‘Hey, you own 30% of the network under these many names,’ they will probably just say ‘sorry’ and move on—or not? At best, they will leave, and we end up losing 30% of the network and quite a few subnets. And if you can’t actually do anything, I wouldn’t be surprised if the node ownership distribution follows a power law, either by competition or even by Dfinity-related entities. @wpb @Lorimer

[Lorimer]

@marcio

At best, they will leave, and we end up losing 30% of the network and quite a few subnets

I can’t imagine it coming to that. I’m not yet convinced that there’s any significant potential for collusion (but I also wouldn’t yet rule it out). I’m simply wanting to raise attention, ask questions and get answers (some are so far being ignored).

If it turns out we do have an extremely high percentage of nodes owned (or indirectly controlled) by 1 entity, the IC Target Topology will need loosening, or alias NPs will need offboarding, or a combination of both. I’m not convinced this will be necessary though. There’s a lot more I’d like to learn about the situation, and that’s all I’m really after at the moment.

In any case, I think it’s important that we’re not shy about addressing these issues, or the IC will never be taken seriously (we shouldn’t try to ignore it for fear of losing some subnets that will have otherwise had a high potential for collusion). Again - I’m not convinced it will come to this.

---

We should be optimising decision making for the long-term success of the IC, not just to avoid short-term hits to confidence. The latter is important, but not at the expense of the former.

---

@Lorimer

• Why are some node providers feeling the need to obscure the terms of their node transfers to other Node Providers by getting them to sign a non-disclosure agreement about the terms of the node transfer?
• How can anyone expect the original NP to be fully relinquishing their claim to those nodes under this sort of circumstance?

[ZackDS]

@Lorimer

If it turns out we do have

There is no such thing as “we” have seen. There is just Dfinity who is now run by Adam.

[marcio]

What I’m struggling to understand is what limits the extent to which a single entity can hoard nodes. The possible reasons I can think of are:

• They aren’t excessively greedy or malicious—perhaps they’re satisfied with controlling 10% rather than 30%. But we could even be dealing (eventually) with North Korea-type actors.
• They believe controlling 10% is significantly less risky than controlling 30%. However, I don’t see how having more nodes would increase the chances of being discovered.
• The process isn’t fully permissionless—for example, DFINITY might have the bias to select certain node providers. But this would still lead to centralization.

IMO, implementing node shuffling and requiring nodes to stake ICP would likely be needed to mitigate this issue.

[Lorimer]

@marcio

IMO, implementing node shuffling and requiring nodes to stake ICP would likely be needed to mitigate this issue.

Node shuffling for sure. I think there are positives and negatives to requiring all NPs to have a large ICP stake. It would significantly limit the rate and amount of nodes that can be onboarded.

However, that doesn’t mean it wouldn’t be fantastic to have staked NPs as an option. A portion of all NPs in each subnet could be required to have a large stake (which could be a factor in boosting their rewards). I suggested this a little while ago, back when this type of conversation was getting little traction.

Subnet Management - General Discussion

Proof-of-stake, and decentralised systems in general, aren’t about guaranteeing the behaviour of individual entities. I agree that the barriers to entry shouldn’t be too high for NPs (or it would hurt scalability). But scalability shouldn’t be prioritised over robustness. I think you could have both by having two tiers of NPs. As longs as more than 1/3 of the nodes in a subnet are backed by a large proof-of-stake, then you have a significantly more robust subnet than one where the nodes are no…

There could even be dedicated subnets that require all nodes to belong to staked NPs. @EnzoPlayer0ne mentioned something like this the other day →

Drain the Swamp

I agree. We should have more dapp specific subnets, where the rules for validators are slightly different. I would love a subnet with 50+ machines where you need to stake in order to participate.

[wpb]

I’m cross posting this comment from @jdcv97 here because it seems like an interesting idea to add to the conversation about policies for onboarding new node providers.

@Jdcv97

I think the one that solves this “anonymity “ is having ICA in charge of recruiting this node providers in person, I mean to onboard this node providers they have to present itself in person, this association will have to take their biometrics, scan their eyes, fingerprints, and ask them to sign in person a contract where they guarantee that they are the person of the identity on the node provider KYC, and that this identity matches with the person doing the biometrics. @Jan @bjoern @samuelburri

We gotta detect where is the potential threat here, I don’t see it to much on node collusion if the nodes have real ID’s, I see a HUGE threat on the validation process of the IDENTITY of this node providers, because there’s no real solution to validate identity or bank documentation presented on the KYC process and if it matches or not the real identity of the person trying to apply.

[Lorimer]

@Jdcv97

having ICA in charge of recruiting this node providers

I think this would be a step backwards. It’s also a solution that doesn’t scale, and doesn’t really belong in the world of crypto (IMO)

[Jdcv97]

ICA exists, just that they don’t have any protocol to follow to validate KYC authenticity. ICA should implement new ways to validate this data.

Otherwise the network would rely just on trust, trust that node providers are not lying about their real identity, and other documentation related to KYC.

The whole point of KYC in @dominicwilliams mind back in the day, when he used to talk and talk about deterministic decentralization, was that this KYC will imply on real world laws against any malicious actor, that’s perfect, but after some
Time I realized, ok but what about if this identities are fake? Which law is gonna apply? To whom?

So now I have a huge concern about this topic, has Dfinity stopped to think about this, or is something new they are just realizing because of me? I really hope they come to this forum with solutions. @Jan

Otherwise we are “fucked” sorry for the expression, but as the IC grows there’s no way no validate if all
The node providers are real or are fake, or most of them are honest or Most of them have lied in the KYC.

This will led to the same issues we tried to avoid on other blockchains, and we will have to implement more and more nodes on the subnets to reduce this threat, compromising scalability, costs, performance, and all
The limitations this other useless blockchains faced because they chose security over scalability.

[sat]

@Lorimer

Node shuffling for sure.

Node shuffling replaces one problem with some other problems. The problems that node shuffling brings are:

• In case of a large subnet, syncing the state would take quite a while (multiple hours to multiple days) and during this time the subnet would have reduced resiliency and performance. We do not want that this is the normal mode of operation.
• In case someone is close to having 1/3 of nodes in the subnet but is not there yet, the shuffling may actually allow them over the 1/3 threshold for a limited amount of time, which may be sufficient to conduct an attack. So we need the right code in place (on chain) to check that decentralization is preserved in this random shuffling.

I am of a view that we need to separate: a) subnet state (data), b) key material, which would allow us to regenerate / rotate the key material without having to delete and rotate the subnet data. Right now the subnet state and key material are bundled together so if you want to rotate the nodes/key material, you HAVE to move data across nodes as well. As we scale the storage capacity of IC nodes over multiple TBs of data, random node shuffling becomes less and less attractive. But all this is a whole other story.

[marcio]

Another option is to allow dApps within a subnet to choose their own providers using allow/reject lists. This would encourage more scrutiny of node providers (NPs), improving transparency and reputation. The downside is that it could lead to greater centralization. However, we could mitigate this by requiring each subnet to have at least 13 distinct, high-reputation NPs. In that case, I could pin a few Dfinity nodes alongside well-known node providers to my subnet.

We could also introduce the option to stake ICP, which would further enhance reputation.

[sat]

@marcio

Another option is to allow dApps within a subnet to choose their own providers using allow/reject lists.

Subnets now have a fixed list of nodes and thus providers. Nodes in a subnet can be replaced with a proposal but a dApp cannot be moved from one subnet from another. This is something that DFINITY just started to look into from the technical side, although this was seen as a major effort earlier.

I presume that once canister migration is implemented, it will effectively be possible for canister (dApp) owners to migrate canisters to the subnets they deem secure or otherwise suitable.

However, it will still not be possible to pick a set of nodes/providers that your canister (dApp) would run on. The set of nodes is fixed for the subnet based on the registry configuration. It can be changed with a proposal, but then it’s fixed again until the next change.

@marcio

We could also introduce the option to stake ICP, which would further enhance reputation.

IMHO this would be a good idea. However, it’s not a full protection. One could stake $100k and be ready to lose them if they would make $10M. We could maybe say to be on subnet X that holds TVL value of $100M, you need to stake at least 1% (or something) of the subnet TVL. But even then, not many NPs would be willing or able to be in such subnets since they wouldn’t be willing to make such a large investment/staking. So decentralization on such “critical” subnets would be at risk.

These are not easy problems to tackle, and we certainly need to discuss them in depth before making decisions. If you have concrete suggestions, please feel free to create a (new) forum thread (one thread per idea), so that we can have a focused discussion.

[Jdcv97]

But the issue and potential threat is on the node provider itself isn’t it? That’s why node shuffling makes sense, because if one subnet (NNS) is being the target of malicious actors, if the same node providers remain the same for 1 month, 1 year, 1 decade whatever, for example, they will always know who to target,
Which data center to attack, which person to go and find physically, what is the point moving key materials or generating new keys if the node provider remains the same on the same subnet. I simply don’t get it.

Doesn’t @Jan have a long video posted on Dfinity youtube where he explains the advantages of chain key and how easy and fast is to onboard a new node to a subnet through “Catch up packages” ? And that this makes the IC so unique because we don’t have to
Download the whole blockchain from genesis? So now this doesn’t work?

[sat]

There is a lot of information to digest about the IC @Jdcv97. It’s a complex system, certainly not easy. A ton of information is scattered across many places (forum/wiki/docs/source code). But please dig in and ask questions if you don’t understand something. That’s the way to learn.

@Jdcv97

That’s why node shuffling makes sense, because if one subnet (NNS) is being the target of malicious actors, if the same node providers remain the same for 1 month, 1 year, 1 decade whatever, for example, they will always know who to target,

Based on the Target Topology or previous version, each NP has zero or one node in the NNS subnet. We don’t have that many NPs with live and healthy nodes, so nearly all NPs are currently represented on the NNS subnet. Therefore, every NP is a good target. Feel free to play with Subnet Management - DRE Documentation to see what the current decentralization is, which NPs are in which subnet, and what could be done to improve decentralization. We have a similar rule about 1 DC and DC provider per subnet.

That said, the best that the tooling can do is use the data it has. So if the input data is wrong (e.g. due to a Sybil attack), then tooling won’t be able to achieve much. This is why Lorimer’s initiative is awesome. Witch hunt is not great though since it may remove some legitimate honest NPs from the subnet and actually make the subnet more vulnerable. We don’t wan’t to do that. No one wants to. So we have to stay calm and act with reason.

@Jdcv97

Which data center to attack, which person to go and find physically, what is the point moving key materials or generating new keys if the node provider remains the same on the same subnet. I simply don’t get it.

Precisely. I see little value in shuffling nodes if we get the same provider(s) into the subnet. And we can’t get new providers because there aren’t that many. And if we shuffle frequently, that means that a malicious NP (if there is one) will be in the subnet from time to time instead of always or never, which might actually make their life easier, since they can now predict that they will be in the subnet at some point. If we had a lot more unique node providers, then I’d view shuffling differently.

@Jdcv97

Doesn’t @Jan have a long video posted on Dfinity youtube where he explains the advantages of chain key and how easy and fast is to onboard a new node to a subnet through “Catch up packages” ? And that this makes the IC so unique because we don’t have to
Download the whole blockchain from genesis? So now this doesn’t work?

We indeed do not need to download the whole blockchain since genesis. HOWEVER, even the latest state of a subnet shared in a CUP can be hundreds of GBs in size, e.g.:
https://dashboard.internetcomputer.org/subnet/3hhby-wmtmw-umt4t-7ieyg-bbiig-xiylg-sblrt-voxgt-bqckd-a75bf-rqe (650GB)
https://dashboard.internetcomputer.org/subnet/2fq7c-slacv-26cgz-vzbx2-2jrcs-5edph-i5s2j-tck77-c3rlz-iobzx-mqe (540GB)

In the future, we can expect some subnets to grow significantly more than that, so easily 10x more than that.

So we can’t count on quick syncs, if a NP is expected to provide ~300Mbit/s (or equivalent of 40MB/s) bandwidth per node. If subnet size is 650GB, then the CUP for the subnet will be a bit over 650000 MB, and divided by 40MB/s you get 4.5 hours of full bandwidth used to sync state/CUP. Then after the 4.5h, the node will have to sync again since it’s now 4.5h behind the rest of the subnet. And then do the same thing a few times more until it fully catches up. And in the meantime, the node is useless, and the provider will get penalties since the node is not productive in this time.

So the math just doesn’t add up. This is not a scalable and reliable way to provide security. Sounds good in theory though. With some changes in the protocol we could make it good in practice as well. But not without changes IMHO.

[Lorimer]

@sat

Node shuffling replaces one problem with some other problems. The problems that node shuffling brings are:

• In case of a large subnet, syncing the state would take quite a while (multiple hours to multiple days) and during this time the subnet would have reduced resiliency and performance. We do not want that this is the normal mode of operation.
• In case someone is close to having 1/3 of nodes in the subnet but is not there yet, the shuffling may actually allow them over the 1/3 threshold for a limited amount of time, which may be sufficient to conduct an attack. So we need the right code in place (on chain) to check that decentralization is preserved in this random shuffling.

I think the first point really just comes down to how frequently you shuffle doesn’t it? If you randomly shuffle one node per subnet per week, surely that’s manageable? The point would be to introduce a slow be steady opposing force against anyone who may be trying to intentionally cluster their nodes within the same subnet. I’m not suggesting that anything like this should be implemented until Subnet Membership changes can be automated on-chain (without requiring an NNS proposal).

I think the second point isn’t a problem specific to node shuffling. I think it’s more likely to occur without shuffling, because if there’s a bad actor who wants their nodes clustered in a subnet, they’ll do their best to introduce biases that cause that to occur, either by steadily submitting proposals themselves, or by influencing when some of their nodes become unassigned (by intentionally degrading them for a short period of time).

I personally think the need for shuffling becomes more obvious when you imagine a future scenario where there are 10s of thousands of nodes and/or NPs, and many, many subnets, some that still have only 13 nodes. Eventually it won’t be possible to micro-manage all of this stuff effectively.

[borovan]

I think if it was the case that everyday you’re shuffling, you could potentially lose your mind.

[Lorimer]

@sat

if we shuffle frequently, that means that a malicious NP (if there is one) will be in the subnet from time to time instead of always or never, which might actually make their life easier, since they can now predict that they will be in the subnet at some point.

I think the probabilities work in favour of node shuffling (at least according to some back of the envelope calculations )

@Lorimer

the probability that a 13 node subnet will be in a state that facilitates collusion (to the point of controlling the subnet) after a shuffle step is 0.1^9. If a shuffle step occurs daily, it would take many many thousands of years for a 24 hour collusion time windows to plausibly open up (by my back-of-the-envelope calculations). Also, shuffling doesn’t need to be a silver bullet, it just needs to be better than not shuffling.

I hope node shuffling is taken under consideration moving forward rather than considered a non-starter. I don’t think shuffling should be made a priority, but I do think it will eventually be needed. I’d love to see it implemented once automated on-chain subnet membership changes are a thing, once the IC has scaled a little more (so there are more NP options), and I guess once/if performance related protocol changes get implemented to reduce catchup latency.

@sat

If we had a lot more unique node providers, then I’d view shuffling differently.

@sat

With some changes in the protocol we could make it good in practice as well

[Lorimer]

I’d love to see node shuffling on the road map. Until then…

LMFAO - Party Rock Anthem ft. Lauren Bennett, GoonRock

[sat]

@Lorimer

I hope node shuffling is taken under consideration moving forward rather than considered a non-starter. I don’t think shuffling should be made a priority, but I do think it will eventually be needed.

Oh it most definitely is being taken under consideration, it has quite a few supporters within DFINITY!

[marcio]

@sat

These are not easy problems to tackle, and we certainly need to discuss them in depth before making decisions. If you have concrete suggestions, please feel free to create a (new) forum thread (one thread per idea), so that we can have a focused discussion.

Basically to do something similar to all other blockchains. This may help:

Proposal: Transforming ICP Node Providers into Validator-Like Entities

Proposal: Transforming ICP Node Providers into Validator-Like Entities Objective Enhance the Internet Computer’s decentralization and security by introducing staking and delegation for node providers, aligning them with Proof-of-Stake (PoS) validator models. Current Model Node providers are pre-approved entities receiving fixed monthly ICP payments from the Network Nervous System (NNS). Proposed Changes 1. Staking and Delegation Node providers stake a minimum amount of ICP (e.g., 50,000 ICP). …