Subnet Management - General Discussion

[sat]

@Lorimer

Thanks @sat, but the upgrade path guarantees are strictly +1/-1 (in terms of version steps).

Answered in the other thread Subnet Management - o3ow2 (Application) - #6 by sat

[timk11]

@sat

This does not necessarily mean that decentralization got worse. For instance, you might be improving in other features (dimensions) such as Country, Continent, etc., while not making the decentralization worse.

Thanks @sat . That’s very helpful documentation indeed. Lots to think about there, and I think I finally got my head around the Nakomoto coefficient! One particular question - Should it be coded in that if a node replacement option would worsen one of the parameters, such as giving an existing node provider a second node in a subnet, then that option should be discarded? Or would this go against other principles involved in this process?

[sat]

@timk11 in some cases, such as when replacing unhealthy nodes, we may need to reduce decentralization. But if there is no other choice … then we need to do it.

[Lorimer]

New ‘Change Subnet Membership’ proposals have been raised:

• uzr34 (currently unclear why certain unassigned nodes haven’t been used)
• pzp6e (looks good, adopted )
• tdb26 (looks good, adopted )
• 5kdm2 (looks good, adopted )
• mpubz (looks good, adopted )

@Sat, can I ask what’s stopping you from announcing these proposals on the forum and providing a link in the proposal summary to the forum discussion? I’m aware this will become a formal requirement in September (NNS proposal discussions - Governance / NNS proposal discussions - Internet Computer Developer Forum (dfinity.org)), but it seems like something that would be easy enough to start doing now wouldn’t it?

Most voters currently won’t know where to look when they see one of these proposals in the NNS dapp. Do you have any objections to providing a link to the forum?

[Lorimer]

New ‘Change Subnet Membership’ proposals have been raised:

• uzr34 looks great, brings this subnet inline with IC target topology limits - adopted
• tdb26 looks great, brings this subnet closer to IC target topology limits - adopted

[Lorimer]

There are 5 new subnet config update proposals that have been raised.

• fuqsr (0 tx/s ?)
• bkfrj
• ejbmu
• csyj4
• 6pbhf

These are the same subnet config update that was applied to the canary subnet last week, which has held a steady and impressive block rate since. More context is available here → Reducing End to End latencies on the Internet Computer

@dsharifi, I expect there’s not a reason to be concerned, but I prefer to ask than to assume. My main thought when I was reviewing the metrics on each of these subnets is that they have very different tx/s and finalisation rate profiles to the canary subnet that this config update has so fair been trailed on. This makes we wonder how representative the results on the canary subnet can be considered. Is there a rush to roll this update out (why roll out to 5 very different subnets in one fell swoop)?

[dsharifi]

The metrics for the canary subnet look very good, and we are confident with rolling out the changes to a new batch of subnets with varying amounts of load. We chose to group subnets in batches with different amounts of load to get a more diverse set of metrics from production on how the subnets behave in production. If there were any regressions, we will discover it much earlier by rolling out to a diverse set of subnets. This is why you correctly found the subnet 6pbhf with many more TX/s than fuqsr both being updated at the same time.

[Lorimer]

Thanks @dsharifi, I’m glad to see all 5 of those subnets doing well post-update.

@dsharifi

• bkfrj → 2.57 Blocks/s
• fuqsr → 2.49 Blocks/s
• ejbmu → 2.45 Blocks/s
• csyj4 → 2.43 Blocks/s
• 6pbhf → 2.10 Blocks/s

While I voted to adopt the first 3 subnets listed above, I voted to reject the last two (the two that were least similar to the canary subnet, with significantly more load and canisters).

My main reason for rejecting was that there was no commentary on these specific proposals from DFINITY (as far as I could see). Only the original announcement, which didn’t relate to any specific proposal, and which didn’t provide any information about the strategy for how this would be rolled out and the testing that’s been performed to inform that strategy (and/or each specific proposal). It’s very difficult (or impossible) to cast an informed vote without information like this. While this sort of information cannot be directly verified, it can be used to cross-reference with other sources of information to gain confidence that everything adds up and seems reasonable.

I had numerous questions about these sorts of details, but didn’t receive an answer until after DFINITY had already voted and the proposal outcomes had been decided.

Would you consider making a few adjustments for the next batch of config updates?

• Please consider providing a reason for why a particular subnet has been chosen for the config deployment (instead of the numerous other subnets that could have been chosen). It would be great if this could also include some details regarding the testing that’s been conducted to inform this decision, and/or the metrics that have been gathered. Without this information you’re asking voters to cast an uninformed vote.

• Please consider declaring the subnet that the proposal relates to in the proposal title (much like the two ‘Change Subnet Membership’ proposals below. Hopefully you can see the difference this makes in navigating the proposals effectively
  

  

• Please consider linking to forum discussion for that particular subnet and/or proposal. (e.g. fuqsr). ‘Change Subnet Membership’ proposals have started doing this recently. Note that the formal URL field doesn’t show up on mobile devices when viewing the proposal in the NNS dapp. The more contextual info / references that can be provided in the proposal summary the better

• Please keep doing the fantastic work that you and the rest of the team are doing to make the IC better and better These performance improvements are very exciting!

[dsharifi]

Hi @Lorimer

I appreciate the thorough feedback! We’ll keep it in mind for the upcoming proposals to update more subnets.

As I mentioned in another thread, we are proposing a gradual rollout of the notarization delay changes, using batches of subnets with varying amounts of load. This is why you correctly observed that subnet 6pbhf with a higher load profile than the canary subnet was proposed by us to be updated.

We have done extensive performance tests with testnets to verify that all these subnets can handle the change, where we have simulated high loads, packet loss between nodes, faulty nodes, etc.

[Lorimer]

@dsharifi

I appreciate the thorough feedback! We’ll keep it in mind for the upcoming proposals to update more subnets

Thank you @dsharifi

I suspect my question over here may come across as niave, but I’m still somewhat confused by those metrics. Would you be able to comment when you get a chance?

[Lorimer]

Latest notorisation delay proposal →

Subnet Management - e66qm (Application)

DFINITY will submit a proposal to reduce the notarization delay on subnet e66qm, similar to what has happened on other subnets in the recent weeks (you can find all details in this forum thread). Note that e66qm is currently under high load from bob.fun, so this proposal could help increase the subnet’s throughput. DFINITY has done tests with the lower notarization delays and high load, so this combination should not cause issues.

[Lorimer]

New proposal for the signing key subnet → Subnet Management - pzp6e (Fiduciary) - Governance / NNS proposal discussions - Internet Computer Developer Forum (dfinity.org)

[Lorimer]

// In general, the round limit should be close to
// `message_limit + 2B * (1 / finalization_rate)` which ensures that
// 1) execution does not slow down finalization.
// 2) execution does not waste the time available per round.
const MAX_INSTRUCTIONS_PER_ROUND: NumInstructions = NumInstructions::new(7 * B);

@Manu, @dsharifi does MAX_INSTRUCTIONS_PER_ROUND need revisiting now that the finalisation rate is being significantly increased on more and more subnets? I’m sure this has already been considered. It would be interesting to hear some commentary Thanks

[Manu]

Yeah we did give it some thought. We think that leaving the round instructions limits unchanged is best. If we would make rounds super short, then the overhead per round (most notably certifying everything) would become a larger fraction of the time, leaving less time for useful computation. So by adjusting the notary delays without adjusting the execution round limits, we will see that if subnets are relatively quiet, we get a high block rate (2+ blocks/s) and therefore lower latency. If subnets are busy, then execution rounds will take ~1s, so the block rate will drop to 1 block/s.

This is nice as it doesn’t reduce the throughput at busy times (which reducing the round limits would) but does improve the latency when it’s not very busy.

[dsharifi]

I created a thread for the w4rem subnet. Subnet Management - w4rem (System Bitcoin)

[Lorimer]

Thanks for explaining this @Manu

[Lorimer]

Thanks for this @dsharifi, and for announcing the new proposals and linking to the announcement from each proposal (I found this very useful while reviewing, in addition to having the subnet name in the proposal titles ).

I’ll list them here just for future reference:

• ejbmu 132395 (duplicate)
• 4zbus 132391
• brlsh 132392
• w4asl 132393
• qxesv 132396
• pae4o 132397
• 2fq7c 132398
• qdvhd 132399
• w4rem 132394 ← The first system subnet to have the notorisation delay reduced (another 13 node subnet though, so 300ms seems right)

This is all very exciting!

---

@Manu, @dsharifi, @andrea, would it be feasible to only specify the payload fields that are necessary in future proposals, or is there a reason for the NULL fields that I’m not considering?

One thing that occurred to me while going through these proposals is that the way subnet config update payloads are constructed is very noisy (meaning there’s a slight danger that there could be a sneaky config change that’s harder to spot, particularly if there are 10 or so other proposals that are submitted at the same time, and only one of them has a subtle deviation).

Given that almost all of the subnet config fields are optional, can I ask why every field is always explicitly included in the payload (with a NULL value). Why not just omit the optional fields that are not being set by the proposal. This would make it very easy to spot exactly what’s changing from a mile off, instead of having to very carefully scan the payload and risk missing something.

[Lorimer]

Expanding the list of Public Subnets on the Internet Computer

Hello IC Community, I’m excited to share some recent developments and propose an update of the list of public subnets on the IC Mainnet. As many of you may have noticed, with the recent introduction of bob.fun, there has been a substantial increase in cycle consumption, surging from approximately 20 TC/s in the last week to over 300 TC/s today. This surge is primarily due to the proof-of-work algorithm as per bob.fun implementation. Subnets are still performing with only a minor impact, and non…

[Lorimer]

Subnet `lhg73` is stalled

Hello everyone. We got alerted that subnet lhg73 is stalled, and we discovered that the nodes in the subnet are in a crash loop. We have identified the root cause and we have already started a subnet recovery process. There is no data loss on the subnet, and it seems to be only a minor problem based on the preliminary analysis, so we expect a straightforward recovery. The whole recovery process should take up to a few hours. We will soon submit a series of proposals. UPDATE: In accordance wit…

[quint]

@Lorimer

@Manu, @dsharifi, @andrea, would it be feasible to only specify the payload fields that are necessary in future proposals, or is there a reason for the NULL fields that I’m not considering?

One thing that occurred to me while going through these proposals is that the way subnet config update payloads are constructed is very noisy (meaning there’s a slight danger that there could be a sneaky config change that’s harder to spot, particularly if there are 10 or so other proposals that are submitted at the same time, and only one of them has a subtle deviation).

Given that almost all of the subnet config fields are optional, can I ask why every field is always explicitly included in the payload (with a NULL value). Why not just omit the optional fields that are not being set by the proposal. This would make it very easy to spot exactly what’s changing from a mile off, instead of having to very carefully scan the payload and risk missing something.

I completely agree with Lorimer’s suggestion. Omitting the optional fields that are not being set would streamline the payloads and make it much easier to spot the actual changes. Reducing noise in the config update proposals would help minimize the risk of overlooking subtle deviations, especially when multiple proposals are submitted at once. It seems like a practical improvement with no downsides, as long as there aren’t any hidden dependencies on NULL values that we might be missing.