Subnet Management - io67a (Application)

[Lorimer]

This topic is intended to capture Subnet Management activities over time for the io67a subnet, providing a place to ask questions and make observations about the management of this subnet.

At the time of creating this topic the current subnet configuration is as follows:

Expand

{
  "version": 44805,
  "records": [
    {
      "key": "subnet_record_io67a-2jmkw-zup3h-snbwi-g6a5n-rm5dn-b6png-lvdpl-nqnto-yih6l-gqe",
      "version": 44805,
      "value": {
        "membership": [
          "2ajl7-n6xum-2j7km-24scw-5tirj-whjlx-756pq-jbcua-utl3q-mts5r-hae",
          "2xph2-xvn4v-pc5xz-3upfj-etpio-qaf2k-d2s3v-jnceu-vcher-qhqav-rqe",
          "cp7d4-uiulo-a46se-yzti4-6qpmz-4rhug-d6rln-7aa5r-nc63x-6q7u7-aae",
          "plbgg-2silg-344nm-n7zqv-lpy4j-uktsq-darh7-shrma-srui4-xrcgp-dae",
          "q3w37-sdo2u-z72qf-hpesy-rgqes-lzflk-aescx-c5ivv-qdbty-s6pgc-jae",
          "tkdjq-rhpgw-7ubus-5w3nx-jrsdi-w4q65-e457d-yt2pt-hsz22-yehmk-uae",
          "xsa4m-ko3b7-2zvcf-56q55-ahyqi-2qkho-obuu3-l4otg-4z6fn-n2r6i-cae",
          "q2ucv-x7dv5-hheao-ocsye-jbg4z-enm75-ss62d-ehqhj-zwwm3-cap5q-tqe",
          "c37f7-3shrz-2mk6g-e2zvp-3z6xc-uuk6r-6yegs-53uzt-c3vtf-dacsv-tae",
          "j63cj-up2z6-xh7m5-m2t5m-t4xi6-6tqz2-ibj4a-mmm2s-7o2bv-ynnc6-xae",
          "ahekl-ihkqy-u4jqo-5kjvc-ygeuv-ejmin-25kld-7vyop-d6unr-uhlrv-wqe",
          "eexw3-cwz3z-6antd-yoyyc-radeu-jmt66-7p2r7-kjbkn-cij4b-kwqrb-tqe",
          "7pwy2-67vvv-agagj-elfj4-gtef2-jxddy-ywxtt-74afr-aeuil-zecky-iae"
        ],
        "nodes": {},
        "max_ingress_bytes_per_message": 2097152,
        "max_ingress_messages_per_block": 1000,
        "max_block_payload_size": 4194304,
        "unit_delay_millis": 1000,
        "initial_notary_delay_millis": 600,
        "replica_version_id": "6968299131311c836917f0d16d0b1b963526c9b1",
        "dkg_interval_length": 499,
        "start_as_nns": false,
        "subnet_type": "verified_application",
        "features": {
          "canister_sandboxing": false,
          "http_requests": true,
          "sev_enabled": false
        },
        "max_number_of_canisters": 120000,
        "ssh_readonly_access": [],
        "ssh_backup_access": [],
        "ecdsa_config": null,
        "chain_key_config": null
      }
    }
  ]
}

[Lorimer]

There’s currently an open ‘Update Subnet Config’ Subnet Management proposal for this subnet → Proposal: 132123 - ICP Dashboard (internetcomputer.org)

Proposal Summary:

This proposal sets the notarisation delay of the subnet to 300ms, down from 600ms. The change will increase the block rate of the subnet, aimed to reduce latency of update calls.

---

600ms (the current configuration) is the default configuration for application subnets.
At the moment there are only two configurations used for initial_notary_delay_millis.

• 1000ms: used by four critical subnets (large subnets) - tdb26 (nns), pzp6e (fiduciary), uzr34 (II), x33ed (SNS)
• 600ms: used by all other subnets (13 nodes subnets)

Larger subnets naturally require larger delays. However @dsharifi announced earlier today plans to increase the block rate, thanks to numerous performance enhancements that have been implemented.

@dsharifi

We also want to increase the rate at which Internet Computer subnets produce blocks. A higher block rate means messages can be included in a block sooner, leading to a lower latency.

@dsharifi are you able to provide some information about why this subnet was chosen to roll out this new config? Presumably eventually all 13 node subnets will use this configuration? What about the larger subnets, will you be aiming to keep the same relative proportions by halving the notary delay to 500ms (or lower)? (I’m only asking because I’m aware that there’s currently a drive to minimise configuration differences between NNS and other subnets).

Thanks in advance

[Lerak]

@Lorimer 300ms is too low for nodes on the other side of the world.

This node is in sc1 data centre, an hour north of Brisbane in Australia. I have been monitoring ping times from a server on AWS in Frankfurt to a node on the io67a subnet q3w37-sdo2u-z72qf-hpesy-rgqes-lzflk-aescx-c5ivv-qdbty-s6pgc-jae :

400ms would be a better number, and a more relavant test, if we want to move all application subnets to a lower number in future.

[Lorimer]

@Lerak

@Lorimer 300ms is too low for nodes on the other side of the world.

Thanks @Lerak. Nice idea to get an indication of network latency. It would be interesting to hear more from the networking team about how representative they consider this to be (@dsharifi).

My understanding is that the initial_notary_delay_millis is the minimum amount of time that all nodes need to wait to notarize (but they can wait longer, with notarisation falling back to other ranked blocks). Setting this somewhere close to the network latency that can be reasonably expected under ideal conditions may be intentional (I’m not sure). If this is set too low for all nodes to be able to disseminate artifacts in time, my understanding is that it’s more likely that multiple notarised blocks at the same height will occur (but the subnet will still be able to make progress, albeit on multiple chains that will need pruning by the finalisation process).

@Manu, are you able to provide some insight into the trade-offs involved in setting the initial_notary_delay_millis for optimal subnet performance? (and worst and best case scenarios that can be expected)

[Lorimer]

I’ve had a few more thoughts after sleeping on this.

@Lorimer

I am curious as to why this parameter is a fixed value rather than adaptive (adaptive to network conditions that ebb and flow - incrementing when network conditions are bad, and decrementing when better performance can be achieved). An implementation like this would have avoided the need to update the config (the notarization delay would have gradually reduced to an equilibrium point that optimises for throughput) - just thinking out loud.

[dsharifi]

Hi @Lerak,

Given that artifacts are pushed with the new P2P implementation, a node only needs 1/2 RTT for its notarization share to reach a peer.

We have done extensive performance tests, where we have simulated the network topology of the io67a subnet by simulating RTTs, packet loss, and bandwidth to mimic production settings.

Of course, if we see any regressions we will propose adjusting the delays again.

[Manu]

Hi @Lerak! I guess you’re thinking about the risk of nodes with higher ping being unable to propose blocks. So in addition to what @dsharifi said, note that this initial notary delay is only the start of when notaries start signing the block from the highest priority block maker. Only after initial_notary_delay + unit_delay (which would be 300 ms initial_notary_delay if this proposal passes + 1000ms unit delay) do notaries fall back to supporting blocks from lower priority block makers.

We will keep an eye on the block making metrics to ensure that no problems are introduced.

[dsharifi]

Hi @Lorimer,

Thanks for creating this new thread specifically for the io67a proposal!

@Lorimer

@dsharifi are you able to provide some information about why this subnet was chosen to roll out this new config? Presumably eventually all 13 node subnets will use this configuration? What about the larger subnets, will you be aiming to keep the same relative proportions by halving the notary delay to 500ms (or lower)? (I’m only asking because I’m aware that there’s currently a drive to minimise configuration differences between NNS and other subnets).

Yes, the goal is indeed for all 13 node subnets to be configured to have an initial notary delay of 300ms.

We don’t have a specific value in place yet for the larger subnets. We are still doing extensive benchmarks to ensure that we find a safe value such that all nodes can make block proposals, and keep up in larger subnets. As we are doing a gradual rollout, we will not propose to adjust the larger subnets until we have adjusted all 13 node Application subnets with metrics to back up that everything works as expected.

[Lorimer]

Thanks @dsharifi, sounds great I’ve voted ‘yes’

[Lerak]

@Manu Yes - I am concerned that it can affect the node’s trustworthy metrics stats. Thus when this node is chosen to be the blockmaker - will there now be an increased probability of timing out?

[Manu]

Theoretically yes, right now the node has to propose its block in 1600 ms, and with this proposal it reduced to 1300 ms, so it’s not a huge reduction. We will keep an eye on the trustworthy node metrics to see if we see any noticeable differences, and please let us know if you notice anything.