Oh this is surely technically possible. However the cost-benefit is a unclear and I’m not convinced it’s worthwhile.
Here is what needs to be done, in more details:
- Create such a big evaluation subnet and create artificial long running workloads. Short-running workloads would not create sufficient impact on the trustworthy metrics since we only keep track of daily stats.
- Creating artificial long-running workloads that exercise a) computation, b) network, c) number of canisters, and d) storage is not a trivial task. Surely can be done, but is not trivial.
- If we don’t exercise all dimensions then node performance in this evaluation subnet will not be as representative as the real performance they will would have in a real subnet
- Next, assuming that we have this big evaluation subnet, we also need to think about the decentralization of the subnet. We currently don’t have a way to make cycles untrusted - cycles are cycles in the current implementation, regardless of where they come from. So if this subnet is not sufficiently decentralized then it could theoretically mint an arbitrary number of cycles and send to other subnets. This can also be prevented with work/development, but we don’t have support for “untrusted subnets”.
- Next, we need to move nodes between the big evaluation subnet and the useful subnets. We currently only have tooling for changing subnet membership of a single subnet. So we would have to submit 2 proposals: 1) remove the nodes that we want to add to a real subnet, and 2) add the nodes. This means 2x the proposals, 2x the work for everyone, and 2x the delay in topology management. Unless we want to change the tooling. Which would be … you guessed it: development
- Finally, although possibly a bit less of an importance but still… somewhat important: servers under load consume more electricity than servers without load. Roughly 2x, observed on some of the existing nodes. Without a good reason to consume that, it’s a bit wasteful.
So yes, it could be done. It’s cost vs benefit.
I don’t see a fundamental problem with extrapolation, although I admit it’s not a perfect representation of the performance of all nodes in a particular DC.
This actually goes both ways:
- DC with “bad” (underperforming) nodes, where some of these bad nodes go into idle subnets and show up as “good” nodes
- DC with “good” nodes, where some of these good nodes go into extremely busy subnets and show up as “bad” nodes
(other combinations of “bad nodes to busy subnets” and “good nodes to idle subnets” can be seen as non-concerning, so do not need to be discussed)
So, arguably, a NP can be unlucky to be in the 2nd group, or lucky and in the 1st group. But statistically, these two should cancel each other, so every NP has a decent chance (75% of cases) of getting appropriate rewards. Not ideal, but not too bad either.
Additionally, this is also something that the NP can improve upon. Every NP can submit proposals to add their nodes to subnets (the tooling is open source), and if that improves decentralization, the community should accept these proposals. So by having nodes in preferable locations or with other preferable characteristics (such as stability), NPs can increase the number of nodes in subnets. And with this, reduce the risk of getting incorrect performance extrapolation.
Sorry for this long post, but seems like you care about the topic so wanted to share my view in details. Thanks for reading this far and hope to get great feedback (as usual from you).