Subnet Management - e66qm (Application)

This topic is intended to capture Subnet Management activities over time for the e66qm subnet, providing a place to ask questions and make observations about the management of this subnet.

At the time of creating this topic the current subnet configuration is as follows:

Expand
{
  "version": 44311,
  "records": [
    {
      "key": "subnet_record_e66qm-3cydn-nkf4i-ml4rb-4ro6o-srm5s-x5hwq-hnprz-3meqp-s7vks-5qe",
      "version": 44311,
      "value": {
        "membership": [
          "inlh6-ii3uy-bpasa-uzngo-xle3x-d2p2a-ylu3h-neikw-tnmwv-wep5x-nae",
          "sbjj4-mggba-56ihm-lf4hg-mjzad-sg6up-lv4os-56vvp-vagid-ieawy-iae",
          "qicnz-k4irc-lorep-ymp33-kx5l4-hnh6u-scxtx-rlwdu-remff-ubqb3-gae",
          "zbzin-kgyio-vai3o-ghyz4-36boi-4tjvv-7fis2-d2mlq-xk7fx-d4udt-xae",
          "fr3jr-74qz3-jzmga-mpt7c-eipd7-numir-hpczg-egnrw-lrrrh-2f7ws-qqe",
          "pzdyu-3fz4j-jukm5-pimui-aqzjl-dlqxo-mpe42-ihxtg-rjzxt-ibdb6-aae",
          "5i7he-lt457-b3ov6-fsi2c-pufn7-4i2mp-zkacf-4yysn-7fhr6-zfxg2-4qe",
          "6ssdj-55z6j-7q72p-vn255-utr5r-r2lgq-iitti-wnchn-rpwos-hjgrt-lqe",
          "lt3wc-g6uvu-v3jcj-qaxlz-nuoen-oof5g-neahy-44h6g-4uqxy-mlri6-iae",
          "txxda-ewevl-saa7x-hj43c-24xcm-i3hjs-j4xxn-2yujg-ptuif-q2ja6-fqe",
          "kjzcx-5a22v-gcxvx-ilfil-mp54w-hlcsh-r4k6n-t36ia-vzoy4-pvedu-cae",
          "joich-t5ted-j3ibc-n7ytv-bomzl-jqimt-7gc72-xujql-xahtn-grunf-dqe",
          "6xfo2-g6moa-wiadg-3nyup-47m5k-ee7rf-wjxij-lv3q5-kqivn-hsmgp-jqe"
        ],
        "nodes": {},
        "max_ingress_bytes_per_message": 2097152,
        "max_ingress_messages_per_block": 1000,
        "max_block_payload_size": 4194304,
        "unit_delay_millis": 1000,
        "initial_notary_delay_millis": 600,
        "replica_version_id": "a3831c87440df4821b435050c8a8fcb3745d86f6",
        "dkg_interval_length": 499,
        "start_as_nns": false,
        "subnet_type": "application",
        "features": {
          "canister_sandboxing": false,
          "http_requests": true,
          "sev_enabled": false
        },
        "max_number_of_canisters": 120000,
        "ssh_readonly_access": [],
        "ssh_backup_access": [],
        "ecdsa_config": null,
        "chain_key_config": null
      }
    }
  ]
}

There’s an open proposal for changing subnet membership - https://dashboard.internetcomputer.org/proposal/131428. This information is presented below:

  • red marker represents a removed node
  • green marker represents an added node
  • highlighted patches represent the country a node sits within

Table
Country Data Center Owner Node Provider Node
--- Germany Munich (mu1) q.beyond Staking Facilities txxda-ewevl-saa7x-hj43c-24xcm-i3hjs-j4xxn-2yujg-ptuif-q2ja6-fqe
+++ India Greater Noida 1 (gn1) Yotta ACCUSET SOLUTIONS wdg4t-afemy-nttou-sdh2u-b6lik-2cuxv-frfoy-wjige-z4qy5-wheg6-kae
Belgium Antwerp (an1) Datacenter United Allusion lt3wc-g6uvu-v3jcj-qaxlz-nuoen-oof5g-neahy-44h6g-4uqxy-mlri6-iae
Canada Toronto 2 (to2) Cyxtera Blockchain Development Labs inlh6-ii3uy-bpasa-uzngo-xle3x-d2p2a-ylu3h-neikw-tnmwv-wep5x-nae
Switzerland Zurich 6 (zh6) Green.ch Sygnum Bank joich-t5ted-j3ibc-n7ytv-bomzl-jqimt-7gc72-xujql-xahtn-grunf-dqe
China HongKong 1 (hk1) Unicom Pindar Technology Limited sbjj4-mggba-56ihm-lf4hg-mjzad-sg6up-lv4os-56vvp-vagid-ieawy-iae
Georgia Vilnius 2 (vl2) Data Inn George Bassadone zbzin-kgyio-vai3o-ghyz4-36boi-4tjvv-7fis2-d2mlq-xk7fx-d4udt-xae
Japan Tokyo 2 (ty2) Equinix Starbase 6xfo2-g6moa-wiadg-3nyup-47m5k-ee7rf-wjxij-lv3q5-kqivn-hsmgp-jqe
Singapore Singapore (sg1) Telin OneSixtyTwo Digital Capital kjzcx-5a22v-gcxvx-ilfil-mp54w-hlcsh-r4k6n-t36ia-vzoy4-pvedu-cae
Slovenia Ljubljana 2 (lj2) Anonstake Anonstake fr3jr-74qz3-jzmga-mpt7c-eipd7-numir-hpczg-egnrw-lrrrh-2f7ws-qqe
Slovenia Maribor (mb1) Posita.si Fractal Labs AG 6ssdj-55z6j-7q72p-vn255-utr5r-r2lgq-iitti-wnchn-rpwos-hjgrt-lqe
Sweden Stockholm 1 (sh1) Digital Realty DFINITY Operations SA pzdyu-3fz4j-jukm5-pimui-aqzjl-dlqxo-mpe42-ihxtg-rjzxt-ibdb6-aae
United States of America (the) Atlanta 2 (at2) Datasite BLP22, LLC 5i7he-lt457-b3ov6-fsi2c-pufn7-4i2mp-zkacf-4yysn-7fhr6-zfxg2-4qe
South Africa Gauteng 3 (jb3) Xneelo Wolkboer (Pty) Ltd qicnz-k4irc-lorep-ymp33-kx5l4-hnh6u-scxtx-rlwdu-remff-ubqb3-gae

The removed node is replaced with a node based in India. I’ve verified that this node is currently unassigned.

The proposal mentions that his change is needed due to the Munich (mu1) data centre being due for decommissioning. See here for more discussion and references.

DFINITY will submit a proposal to reduce the notarization delay on subnet e66qm, similar to what has happened on other subnets in the recent weeks (you can find all details in this forum thread).

Note that e66qm is currently under high load from bob.fun, so this proposal could help increase the subnet’s throughput. DFINITY has done tests with the lower notarization delays and high load, so this combination should not cause issues.

5 Likes

This proposal is now submitted: https://dashboard.internetcomputer.org/proposal/132368

5 Likes

Thanks for this announcement @Manu. Looks good and I’ve voted to adopt. It’ll be interesting to see the updated metrics once this proposal executes :+1:

2 Likes

There appear to be confounding variables. More commentary here →

It did roughly double the computation rate of the subnet (jumping from ~11B to ~22B instructions per second)

1 Like

Maybe to add a bit of context to my post: When I wrote the post, we had observed messages between subnets that were enqueued for multiple hours. Coincidentally, those messages were all delivered while we were still analyzing the situation (without us doing anything actively). The delays seem to be related to the large number of DTS executions outstanding on the e66qm subnet, caused by the large number of bob miners that all want to run heavy computation. We will analyze this further and provide a more detailed update.

3 Likes

so what was the problem with the high load from bob? problematic subnet?

Proposal 133445

2 removed nodes replaced with nodes in South Korea and Switzerland. 1 because it’s down (hence the proposal), and the other to improve decentralisation. Looks great, I’ve voted to adopt.

Decentralisation Stats

Subnet node distance stats (distance between any 2 nodes in the subnet) →

Smallest Distance Average Distance Largest Distance
EXISTING 0 km 7144.537 km 15453.375 km
PROPOSED 489.837 km 7472.928 km (+4.6%) 15453.375 km

This proposal increases decentralisation, considered purely in terms of geographic distance (and therefore there’s a slight theoretical increase in localised disaster resilience). :+1:

Subnet characteristic counts →

Continents Countries Data Centers Owners Node Providers
EXISTING 4 12 13 13 13
PROPOSED 4 13 (+7.7%) 13 13 13

This proposal improves decentralisation in terms of jurisdiction diversity. :+1:

Largest number of nodes with the same characteristic (e.g. continent, country, data center, etc.) →

Continent Country Data Center Owner Node Provider
EXISTING 6 2 1 1 1
PROPOSED 5 (-16.67%) 1 (-50%) 1 1 1
:star_struck: :point_up:

See here for acceptable limits → Motion 132136

The above subnet information is illustrated below, followed by a node reference table:

Map Description
  • Red marker represents a removed node (transparent center for overlap visibility)
  • Green marker represents an added node
  • Blue marker represents an unchanged node
  • Highlighted patches represent the country the above nodes sit within (red if the country is removed, green if added, otherwise grey)
  • Light grey markers with yellow borders are examples of unassigned nodes that would be viable candidates for joining the subnet according to formal decentralisation coefficients (so this proposal can be viewed in the context of alternative solutions that are not being used)

Table
Continent Country Data Center Owner Node Provider Node Status
--- Europe Switzerland Zurich 6 (zh6) Green.ch Sygnum Bank joich-t5ted-j3ibc-n7ytv-bomzl-jqimt-7gc72-xujql-xahtn-grunf-dqe DOWN
--- Europe Slovenia Ljubljana 2 (lj2) Anonstake Anonstake fr3jr-74qz3-jzmga-mpt7c-eipd7-numir-hpczg-egnrw-lrrrh-2f7ws-qqe UP
+++ Europe Switzerland Zurich 4 (zh4) Nine.Ch Tomahawk.vc 5oe2d-on34f-iyvla-5ejfo-34n5u-yllm4-zxdgu-scaji-uleio-yeszd-yqe UNASSIGNED
+++ Asia Korea (the Republic of) Seoul 2 (kr2) Gasan Web3game l2bzb-vpnu6-jlsae-poddg-5hpnz-bx54x-3khc2-aazpn-bop4m-zforf-uqe UNASSIGNED
Europe Belgium Antwerp (an1) Datacenter United Allusion lt3wc-g6uvu-v3jcj-qaxlz-nuoen-oof5g-neahy-44h6g-4uqxy-mlri6-iae UP
Americas Canada Toronto 2 (to2) Cyxtera Blockchain Development Labs inlh6-ii3uy-bpasa-uzngo-xle3x-d2p2a-ylu3h-neikw-tnmwv-wep5x-nae UP
Asia China HongKong 1 (hk1) Unicom Pindar Technology Limited sbjj4-mggba-56ihm-lf4hg-mjzad-sg6up-lv4os-56vvp-vagid-ieawy-iae UP
Asia India Greater Noida 1 (gn1) Yotta ACCUSET SOLUTIONS wdg4t-afemy-nttou-sdh2u-b6lik-2cuxv-frfoy-wjige-z4qy5-wheg6-kae UP
Asia Japan Tokyo 2 (ty2) Equinix Starbase 6xfo2-g6moa-wiadg-3nyup-47m5k-ee7rf-wjxij-lv3q5-kqivn-hsmgp-jqe UP
Europe Lithuania Vilnius 2 (vl2) Data Inn George Bassadone zbzin-kgyio-vai3o-ghyz4-36boi-4tjvv-7fis2-d2mlq-xk7fx-d4udt-xae UP
Asia Singapore Singapore (sg1) Telin OneSixtyTwo Digital Capital kjzcx-5a22v-gcxvx-ilfil-mp54w-hlcsh-r4k6n-t36ia-vzoy4-pvedu-cae UP
Europe Slovenia Maribor (mb1) Posita.si Fractal Labs AG 6ssdj-55z6j-7q72p-vn255-utr5r-r2lgq-iitti-wnchn-rpwos-hjgrt-lqe UP
Europe Sweden Stockholm 1 (sh1) Digital Realty DFINITY Operations SA pzdyu-3fz4j-jukm5-pimui-aqzjl-dlqxo-mpe42-ihxtg-rjzxt-ibdb6-aae UP
Americas United States of America (the) Atlanta 2 (at2) Datasite BLP22, LLC 5i7he-lt457-b3ov6-fsi2c-pufn7-4i2mp-zkacf-4yysn-7fhr6-zfxg2-4qe UP
Africa South Africa Gauteng 3 (jb3) Xneelo Wolkboer (Pty) Ltd qicnz-k4irc-lorep-ymp33-kx5l4-hnh6u-scxtx-rlwdu-remff-ubqb3-gae UP

Known Neurons to follow if you're too busy to keep on top of things like this

If you found this analysis helpful and would like to follow the vote of the LORIMER known neuron in the future, consider configuring LORIMER as a followee for the Subnet Management topic.

Other good neurons to follow:

  • Synapse (follows the LORIMER and CodeGov known neurons for Subnet Management, and is a generally well informed known neuron to follow on numerous other topics)

  • CodeGov (actively reviews and votes on Subnet Management proposals, and is well informed on numerous other technical topics)

  • WaterNeuron (the WaterNeuron DAO frequently discuss proposals like this in order to vote responsibly based on DAO consensus)

1 Like

Voted to adopt proposal 133445.

Again Sygnum Bank 's joich node is offline and needs to be replaced.
Adding Tomahawk.vc 's 5oe2d node improves decentralization.

Voted to adopt proposal 133445.

This proposal replaces 2 nodes in subnet e66qm including joich, which appears as “Status: Offline” in the dashboard. As shown in the proposal and verified using the DRE tool, decentralisation is improved with respect to country and all parameters remain within the requirements of the target topology.

Voted to adopt proposal 133445. The proposal replaces two nodes on the e66qm subnet: dead node joich (Offline) and node fr3jr (Active) with nodes l2bzb and 5oe2d. The node joich is healthy and only replaced in order to improve the Nakamoto Coefficient for the Country metric as verified with the Dre tool.

Voted to adopt proposal 134039, as the reasoning is sound and the proposal description matches the payload.

This proposal is intended to replace a dead node, l2bzb, which node appears as “Status: Offline” on the dashboard. As seen in the proposal (which I verified using the DRE tool), the proposed change leaves the target topology parameters unchanged and within the requirements.

2 Likes

Proposal 134039

TLDR: 1 offline node in South Korea replaced with an unassigned node in Australia. This proposal also slightly increases decentralisation in terms of geographic distance and also continent diversity. Looks good, I’ve adopted.

Decentralisation Stats

Subnet node distance stats (distance between any 2 nodes in the subnet) →

Smallest Distance Average Distance Largest Distance
EXISTING 489.837 km 7472.928 km 15453.375 km
PROPOSED 489.837 km 8303.16 km (+11.1%) 16759.085 km (+8.4%)

This proposal slightly increases decentralisation, considered purely in terms of geographic distance (and therefore there’s a slight theoretical increase in localised disaster resilience). :+1:

Subnet characteristic counts →

Continents Countries Data Centers Owners Node Providers Node Operator
EXISTING 4 13 13 13 13 13
PROPOSED 5 (+20%) 13 13 13 13 13

This proposal slightly improves decentralisation in terms of jurisdiction diversity. :+1:

Largest number of nodes with the same characteristic (e.g. continent, country, data center, etc.) →

Continent Country Data Center Owner Node Provider Node Operator
EXISTING 5 1 1 1 1 1
PROPOSED 5 1 1 1 1 1

See here for acceptable limits → Motion 132136

The above subnet information is illustrated below, followed by a node reference table:

Map Description
  • Red marker represents a removed node (transparent center for overlap visibility)
  • Green marker represents an added node
  • Blue marker represents an unchanged node
  • Highlighted patches represent the country the above nodes sit within (red if the country is removed, green if added, otherwise grey)
  • Light grey markers with yellow borders are examples of unassigned nodes that would be viable candidates for joining the subnet according to formal decentralisation coefficients (so this proposal can be viewed in the context of alternative solutions that are not being used)

Table
Continent Country Data Center Owner Node Provider Node Operator Node Status Performance
--- Asia Korea (the Republic of) Seoul 2 (kr2) Gasan Web3game 5dwhe l2bzb-vpnu6-jlsae-poddg-5hpnz-bx54x-3khc2-aazpn-bop4m-zforf-uqe DOWN :bar_chart:
+++ Oceania Australia Melbourne 2 (mn2) NEXTDC Icaria Systems Pty Ltd l5lhp o2q5i-kszma-radgr-fdzu5-iqfps-w6xfc-aka7m-fftq4-2lpar-wj6xc-mqe UNASSIGNED :bar_chart:
Europe Belgium Antwerp (an1) Datacenter United Allusion pgunx lt3wc-g6uvu-v3jcj-qaxlz-nuoen-oof5g-neahy-44h6g-4uqxy-mlri6-iae UP :bar_chart:
Americas Canada Toronto 2 (to2) Cyxtera Blockchain Development Labs 4lp6i inlh6-ii3uy-bpasa-uzngo-xle3x-d2p2a-ylu3h-neikw-tnmwv-wep5x-nae UP :bar_chart:
Europe Switzerland Zurich 4 (zh4) Nine.Ch Tomahawk.vc paxme 5oe2d-on34f-iyvla-5ejfo-34n5u-yllm4-zxdgu-scaji-uleio-yeszd-yqe UP :bar_chart:
Asia China HongKong 1 (hk1) Unicom Pindar Technology Limited vzsx4 sbjj4-mggba-56ihm-lf4hg-mjzad-sg6up-lv4os-56vvp-vagid-ieawy-iae UP :bar_chart:
Asia India Greater Noida 1 (gn1) Yotta ACCUSET SOLUTIONS slaxf wdg4t-afemy-nttou-sdh2u-b6lik-2cuxv-frfoy-wjige-z4qy5-wheg6-kae UP :bar_chart:
Asia Japan Tokyo 2 (ty2) Equinix Starbase dpt4y 6xfo2-g6moa-wiadg-3nyup-47m5k-ee7rf-wjxij-lv3q5-kqivn-hsmgp-jqe UP :bar_chart:
Europe Lithuania Vilnius 2 (vl2) Data Inn George Bassadone inluf zbzin-kgyio-vai3o-ghyz4-36boi-4tjvv-7fis2-d2mlq-xk7fx-d4udt-xae UP :bar_chart:
Asia Singapore Singapore (sg1) Telin OneSixtyTwo Digital Capital d4bin kjzcx-5a22v-gcxvx-ilfil-mp54w-hlcsh-r4k6n-t36ia-vzoy4-pvedu-cae UP :bar_chart:
Europe Slovenia Maribor (mb1) Posita.si Fractal Labs AG 3xiew 6ssdj-55z6j-7q72p-vn255-utr5r-r2lgq-iitti-wnchn-rpwos-hjgrt-lqe UP :bar_chart:
Europe Sweden Stockholm 1 (sh1) Digital Realty DFINITY Operations SA lgp6d pzdyu-3fz4j-jukm5-pimui-aqzjl-dlqxo-mpe42-ihxtg-rjzxt-ibdb6-aae UP :bar_chart:
Americas United States of America (the) Atlanta 2 (at2) Datasite BLP22, LLC 5syyj 5i7he-lt457-b3ov6-fsi2c-pufn7-4i2mp-zkacf-4yysn-7fhr6-zfxg2-4qe UP :bar_chart:
Africa South Africa Gauteng 3 (jb3) Xneelo Wolkboer (Pty) Ltd ymenq qicnz-k4irc-lorep-ymp33-kx5l4-hnh6u-scxtx-rlwdu-remff-ubqb3-gae UP :bar_chart:

Known Neurons to follow if you're too busy to keep on top of things like this

If you found this analysis helpful and would like to follow the vote of the LORIMER known neuron in the future, consider configuring LORIMER as a followee for the Subnet Management topic.

Another good neuron to follow is Synapse (follows the LORIMER and CodeGov known neurons for Subnet Management, and is a generally well informed known neuron to follow on numerous other topics)

1 Like

Voted to adopt proposal 134039. The proposal replaces one nodes from subnet e66qm:
Removed Node: l2bzb, Dashboard Status Offline
Added Node: o2q5i.
The proposal was verified using the DRE tool to verify the metrics stated and there was no impact on decentralization.

1 Like

Voted to adopt proposal #134039.

The proposal is correct in that it replaces offline dead node from Seul2 l2bzb , with Melbourne2 node o2q5i while improving decentralization.

1 Like