Boundary Node Roadmap

Roadmap
81

Hello everyone,

It’s time for another update from the boundary nodes team. With the new roadmap, boundary nodes have their own milestone: “SOLENOID.” This milestone encompasses the new boundary node architecture and its building blocks. Here’s where we stand today:

Progress:

First API Boundary Node Incoming

Most aspects of the API boundary nodes are complete. ic-boundary has been running in production for over six months, there are enough nodes with IPv4 connectivity, and the integration into the NNS and guestOS is finished. We just have some final touches left and are awaiting the final green light from the security review.

Once approved, we will propose adding firewall rules to allow metric and log scraping, followed by a proposal to deploy the first node. We aim to complete this in June, so keep an eye out for the proposal and be ready to vote!

Discovery Library

We are finalizing the implementation of the discovery library and gradually integrating it into agent-rs. This library is crucial for all IC-native clients that need to communicate directly with the API boundary nodes. For example, the HTTP gateway will be a key user of this library.

ic-gateway

We have made significant progress on the HTTP gateway core service. After some testing, we plan to integrate it into the current boundary nodes to “battle-test” it and gather insights just as we did with ic-boundary. Gradually, we will transition to the new HTTP gateways and retire the existing boundary nodes.

We will continue to keep you updated on our progress. We look forward to your feedback and are here to address any questions you may have!

6 Likes
82

Crossposting for completeness :slight_smile: :

2 Likes
83

We are thrilled to announce that the decentralization steps within the SOLENOID milestone are making steady progress. With the adoption of the proposal 130337, the first two API boundary nodes (API BNs) came up live under the management of the NNS and became an integral part of the protocol. Now for the first time ever, IC clients can route their API canister calls directly through these nodes.

While this marks the first big step of the SOLENOID milestone, we are still in the early stages. Below we explain how one can discover the API BNs and send http requests through them.

How can I discover existing API Boundary Nodes?

As API BNs are now part of the IC, they are marked in the registry canister and are also exposed in the system state tree. The following example in Rust allows to fetch all existing API boundary nodes:

/ic$ cargo run --bin fetch_api_bns ic0.app

API Boundary Nodes in the State Tree: [
    ApiBoundaryNode {
        domain: "br1-dll01.aviatelabs.co",
        ipv6_address: "2001:920:401a:1710:6801:94ff:fe18:6b54",
        ipv4_address: Some(
            "213.246.205.106",
        ),
    },
    ApiBoundaryNode {
        domain: "bc1-dll02.blockchaindevlabs.com",
        ipv6_address: "2001:438:8000:1d:6801:c1ff:feac:bc9",
        ipv4_address: Some(
            "208.184.66.5",
        ),
    },
]

Internally a read state call to https://ic0.app/api/v2/subnet/subnet_id/read_state is made asking for the path /api_boundary_nodes in the state tree. All API BNs with their domain names, IPv6 addresses and optional IPv4 addresses are listed under this path.

In the following, we show how one can make use of this information and send requests to an API BN directly.

What can I do with these API boundary nodes?

Now as we’ve discovered the current set of API BNs, we can send API canister calls directly to the IC, thus avoiding the centralized DNS-based routing via ic0.app. Run the previous example with an API BN domain name instead:

/ic$ cargo run --bin fetch_api_bns br1-dll01.aviatelabs.co

The response should of course remain identical. However, now we talk to the IC directly via one of the API BNs https://br1-dll01.aviatelabs.co/api/v2/subnet/subnet_id/read_state.

What’s next?
Soon we’ll extend agent-rs with the ability to automatically discover API BNs and dynamically route traffic to them based on some predefined strategies (e.g., minimizing latency). This would mark another important step towards decentralization.

10 Likes
84

Excellent news!
There is a lot more to say in the way of praise and encouragement and forward planning but “excellent” will do for now :slightly_smiling_face:

3 Likes
85

This is exciting!

However, I’m having some issues with communicating with one of the API BNs.

This command works fine, returning the result after ~2 seconds.

This command instead, after ~10 seconds, returns:

Error: An error happened during communication with the replica: error sending request for url (https://br1-dll01.aviatelabs.co/api/v2/status)

Although, this command works, returning the result after ~2 seconds:

/ic$ cargo run --bin fetch_api_bns bc1-dll02.blockchaindevlabs.com

So, I believe there’s something that doesn’t work properly on the API BN at br1-dll01.aviatelabs.co.

86

Hey Luca,

yes, the one node unfortunately is having some issues (unrelated to being an API boundary node). You can see the status here: https://dashboard.internetcomputer.org/node/ag4ro-4ihpg-ozp6d-lbwac-5pksl-3wany-aaayw-ibvuw-uh7wj-lbav5-qae

We have already submitted a proposal to remove this node here and another proposal to add two more API boundary nodes (here).

Sorry for the hiccups.

2 Likes
87

Hey Luca and thanks for your message! In the world of decentralized API boundary nodes, the health of each individual node is indeed the responsibility of the respective node provider. This time, you encountered an unhealthy node by interacting with it directly. In the near future we’ll extend the agent-rs to route traffic through all dynamically discovered API BNs. The agent will also monitor API BNs health and select the node with minimal latency, thus substantially reducing the probability of dispatching requests to unhealthy nodes. If a request for a certain node still fails by some reason, the agent will automatically retry it with another node, ensuring an overall smooth user experience.

5 Likes
88

Hello everyone,

Right before the end of the week, we are back with some news from the boundary node team.

Progress:

API Boundary Nodes

Since the beginning of June, we have now several API boundary nodes running on mainnet. So far, they have not received much traffic because everything is still routed through the “old” boundary nodes, but that will change. We have mostly just made sure everything runs smoothly: the API boundary nodes come up healthy, they automatically obtain the necessary certificates from Let’s Encrypt and successfully route the traffic. Also, they have already undergone several upgrades.

Discovery Library

The implementation of the discovery library has been wrapped up and we are close to integrating it into agent-rs (here you can find the feature branch). We have already been testing it ourselves and had no issues so far.

HTTP Gateway

Probably the biggest news is the open sourcing of ic-gateway, the core service of the HTTP gateways. You can find the code in the dfinity/ic-gateway repository. This marks again a major milestone in our work towards the new boundary node architecture. As of now, we are not yet working on “stand-alone” HTTP gateways, but we have replaced nginx and icx-proxy with ic-gateway in the old boundary nodes (see the following PR for details). We have had a canary out for testing for almost two weeks now (see the forum post) and we are gradually switching traffic over to the ic-gateway boundary nodes.

Outlook:

As we are getting closer to having all the pieces of the new boundary node architecture in place, we are focusing on getting it ready for production by making sure we have all the monitoring, all the dashboards, all the alerts set up and also have the necessary processes for disaster recovery in place.

We will continue to keep you updated on our progress and look forward to hearing your feedback and addressing any questions you may have!

8 Likes
89

Hello everyone,

It is time for yet another update from the boundary nodes.

Progress:

HTTP Gateway

After having open-sourced ic-gateway, we switched the “old” boundary nodes to ic-gateway, effectively replacing nginx and icx-proxy. These boundary nodes have now been running in production for more than two weeks without any hiccups.

As we are gaining more experience and confidence, we are working on the “stand-alone” HTTP gateways, which we will start testing soon.

Maintenance

Besides that, we have invested quite some time in performance and reliability improvements of ic-boundary and ic-gateway. As part of that, we extracted the common “features” into a shared library, called ic-bn-lib (here is the code).

Outlook:

Currently, we are in the process of getting the “stand-alone” HTTP gateways ready for production. And we are continuing to wrap up a few things required for production: monitoring, dashboards, alerting, and disaster recovery. More on that soon!

We will keep you updated and look forward to hearing your feedback and addressing any questions you may have!

7 Likes