As we just submitted another two API boundary node management proposal, I realized that it is probably better to have a single thread for all these proposals instead of creating a new one every time.
We submitted two proposals to swap out three nodes that have been cordoned. The proposal #134993 propose to add three replacement nodes for the three nodes that are proposed to be removed in proposal #134994.
Reason:
The proposals are correct, they replace 3 API boundary nodes that have been cordoned, xzsfe from TY2 DC in Tokyo, k3b33 from TO1 DC in Toronto and yncop from CH2 DC in Chicago with unassigned healthy nodes coqzx from KR2 DC in Korea, jpduk from JV1 DC in Florida and lvoro from AT2 DC in Georgia.
CodeGov has a team of developers who review and vote independently on the following proposal topics: IC-OS Version Election, Protocol Canister Management, Subnet Management, Node Admin, and Participant Management. The CodeGov NNS known neuron is configured to follow our reviewers on these topics and Synapse on most other topics. We strive to be a credible and reliable Followee option that votes on every proposal and every proposal topic in the NNS. We also support decentralization of SNS projects such as WaterNeuron and KongSwap with a known neuron and credible Followees.
Learn more about CodeGov and its mission at codegov.org.
The proposals are set to replace 2 API boundary nodes :
One dead offline status node lvoro from AT2 DC in Atlanta, and second Degraded status node dkadh from same DC and NP.
Now the question is shouldn’t the 2 replacement nodes be available as unassigned and Awaiting ?
You are completely right about the replacement nodes. It looks like the chosen nodes got redeployed (node ID changes) and are not available anymore. Hence, the proposal is “useless”. I propose to accept the “removal” proposal as the nodes are anyway currently dead and we will submit another “addition” proposal to add two alive nodes.
Makes sense, as a follow up any reason they were redeployed, as in how to avoid this for future replacement ? NP Rivonia has 13 nodes unassigned in DC NY1 and also one offline node in subnet pzp6e so should be plenty to choose from.
Proposal: 135310
Vote: REJECT
Comments: This reject was at the recommendation of one of our CodeGov team members, @ZackDS.
Proposal: 135311
Vote: ADOPT
Comments: This adopt was at the recommendation of one of our CodeGov team members, @ZackDS. He reviewed and verified the proposal is acceptable.
About CodeGov
CodeGov has a team of developers who review and vote independently on the following proposal topics: IC-OS Version Election, Protocol Canister Management, Subnet Management, Node Admin, and Participant Management. The CodeGov NNS known neuron is configured to follow our reviewers on these technical topics. We also have a group of Followees who vote independently on the Governance and the SNS & Neuron’s Fund topics. We strive to be a credible and reliable Followee option that votes on every proposal and every proposal topic in the NNS. We also support decentralization of SNS projects such as WaterNeuron, KongSwap, and Alice with a known neuron and credible Followees.
Learn more about CodeGov and its mission at codegov.org.
Currently, redeployments are ongoing as gen1 NPs have to reduce the total number of nodes they have to 42 or less. Hence, nodes have been sold and are being off- and onboarded by the new owners. This is just my guess and I don’t know whether this is the case for NY1.
The problem is that not every node is suitable to be an API boundary node. Nodes need to be deployed with an IPv4 address and a domain name. You will see that in the node record:
$ ./ic-admin --nns-url https://icp-api.io get-node 4fssn-4vi43-2qufr-hlrfz-hfohd-jgrwc-7l7ok-uatwb-ukau7-lwmoz-tae
Using NNS URLs: ["https://icp-api.io/"]
Fetching the most recent value for key: node_record_4fssn-4vi43-2qufr-hlrfz-hfohd-jgrwc-7l7ok-uatwb-ukau7-lwmoz-tae
Most recent version is 48125. Value:
Node { xnet: Some("[2a00:fa0:3:0:6801:4dff:fee5:e3d2]:2497"), http: Some("[2a00:fa0:3:0:6801:4dff:fee5:e3d2]:8080"), node_operator_id: 5atxd-q4jom-dtpxr-awd3p-e562x-hpavj-godtj-g6vmz-of2c6-ildhh-fae, chip_id: None, hostos_version_id: Some("2e269c77aa2f6b2353ddad6a4ac3d5ddcac196b1"), public_ipv4_config: Some("ip_addr 83.137.84.35 gw [\n \"83.137.84.33\",\n] prefix_length 28"), domain: Some("node2.ge2.extragone.com"), node_reward_type: Some("type1") }
Most NPs have per DC two nodes that meet the requirements.
Thanks, this is valuable info about the min 2 nodes with ipv4 and domain name for Boundary use per NP. Another question would be if there is a minimum number of Boundary Nodes on standby, or it depends on traffic per region ?
After checking for ipv4, it looks like none of the nodes from this NP is suited for replacement, and there is currently no other NP on the NY1 DC.
Some background/context:
The API BNs are all active and ready to serve requests (minus the ones that have issues such as broken network connectivity or a hardware failure). In order to increase capacity, additional unassigned nodes need to be turned into API BNs through a proposal.
At the moment, capacity is not an issue at all as there are similar number of API BNs as there were legacy BNs and compared to the legacy BNs, the API BNs are extremely powerful since they run on node machines.
The most important aspect at the moment is geographical coverage and fault tolerance: in order to minimize user-experienced latency, there should always be an API BN close by. In addition, if a API BN fails, there should be an alternative relatively close-by.
With our initial proposal, we proposed to cover the same regions as the legacy BNs did plus some extra where a node was available (e.g., Africa).
This is however not that easy as only about 80-90 out of all nodes actually have an IPv4 address and a domain name. 18 are currently in use as API BNs, 24 are assigned to a subnet and a bit more than 40 are unassigned and available to be used.
This seems like a problem. Are there any plans or future node provider incentives to increase the number of nodes with an IPv4 address and a domain name? It seems like we should make sure there are suitable API BN everywhere including in the most population dense regions such as NY1.
At the moment, NPs are asked to deploy at least two nodes per DC with IPv4 and a domain name. This ensures that there are API BNs in every location, where there are also replicas. At the moment, with all the redeployments ongoing right now, this is not always ensured.
Hey @alexu@sat is there any reason why verification that a node provider has 2 nodes per DC with IPv4 connection and a domain name shouldn’t be part of the verification for onboarding and/or a prerequisite for subnet assignment and receipt of remuneration? Hasn’t this requirement for IPv4 been in place for a long time? It sounds like a lack of the availability of multiples of these types of nodes in each DC can lead to end user performance problems.
This is certainly something that can be checked during subnet management reviews, particularly if this is seen as a critical requirement. My impression is that this is about reducing request latency as much as possible (by aiming for there to be a local replica for each boundary node to route a request to). Please correct me if I have the wrong end of the stick.
One question is whether it should be prioritised over decentralisation metrics and existing business rules. I’d expect not and that this is more of a nice to have.
I’ll take a look at running some analysis when I get a chance, and potentially encorporate this into future subnet management reviews. @aligatorr89, @MalithHatananchchige, you may also find this interesting
Although the main intention to require a minimum of 2 API BNs per DC is to minimize latency in requests to local replicas, I think the problem being discussed is if there’s no checking of this requirements we might end up with DCs with no API BNs which means that the user as to make a request to a node much further away which increases latency and translates into poor UX.
As it is established right now, all API BNs have close by local replicas to make requests, the problem is how far the API BNs are from the users.
Proposal 135446 turns three nodes into an API boundary node to replace the two nodes that were removed last week (Proposal 135310) and the one node that will be removed in the above proposal.
Second, about the API BN location and the routing. At the moment, it mostly matters that there are API BNs all over the world such that they are close by the clients and the HTTP gateways. The request routing from the API BNs to the replicas mostly works in a round-robin manner. Hence the location doesn’t matter that much. For more information, you can check my post here.
Reason:
The proposal removes the Degraded status (with IC_PrometheusTargetMissing) API Boundary node jpduk from the JV1 DC in Jacksonville,Florida currently the only one API Boundary node under NP Rivonia Holdings LLC.
About CodeGov
CodeGov has a team of developers who review and vote independently on the following proposal topics: IC-OS Version Election, Protocol Canister Management, Subnet Management, Node Admin, and Participant Management. The CodeGov NNS known neuron is configured to follow our reviewers on these technical topics. We also have a group of Followees who vote independently on the Governance and the SNS & Neuron’s Fund topics. We strive to be a credible and reliable Followee option that votes on every proposal and every proposal topic in the NNS. We also support decentralization of SNS projects such as WaterNeuron, KongSwap, and Alice with a known neuron and credible Followees.
Learn more about CodeGov and its mission at codegov.org.
Reason:
The proposal aims to turn 3 healthy Awaiting status nodes into API Boundary ones. All 3 are from the same NP Giant Leaf, LLC namely:
node ek6px from the AT2 DC in Atlanta, Georgia and 2 from the DC OR1 DC in Orlando, Florida:
node mj4qw and atrq6 that will serve as replacement following the removal of API Boundary nodes from previous proposals that were all in the US East region.
About CodeGov
CodeGov has a team of developers who review and vote independently on the following proposal topics: IC-OS Version Election, Protocol Canister Management, Subnet Management, Node Admin, and Participant Management. The CodeGov NNS known neuron is configured to follow our reviewers on these technical topics. We also have a group of Followees who vote independently on the Governance and the SNS & Neuron’s Fund topics. We strive to be a credible and reliable Followee option that votes on every proposal and every proposal topic in the NNS. We also support decentralization of SNS projects such as WaterNeuron, KongSwap, and Alice with a known neuron and credible Followees.
Learn more about CodeGov and its mission at codegov.org.
The CodeGov known neuron has voted to adopt these two proposals based on the review provided by @ZackDS. This is a proposal topic that is not covered by the Grants for Voting Neurons, so the work that Zack put into these reviews is volunteer. This demonstrates his strong commitment to the decentralization of the internet computer. Thank you Zack.
