CLARIFICATION: Routing behaviour based on geographic proximity between network participants

Developers
1

Quick question, based on a discussion we were having with @frederikrothenberger yesterday.

Do requests made from a certain geographic location

  • get routed to the closest boundary node geographically?
  • does the boundary node receiving the request route the request to the geographically closest replica? Or is this round robin?
2 Likes
2

Hello @saikatdas0790

There are four boundary node regions (US west, US east, Europe, Asia). The DNS loadbalancer will give you the IPs of the boundary nodes in your region.
Your browser (or whatever client you are using) will choose one of them and send the traffic there.
That boundary node then randomly chooses one of the replicas and forwards the request to that replica. Hence, the observed RTT can vary greatly from 10ms to 400ms.

And as an addendum: Why is it important to route to a random replica and not the geographically closest?
The IC can tolerate up to 1/3 of malicious nodes. If the boundary node would always route to the geographically closest replica, it might always route to a malicious node and the client had no straightway to “talk” to a benign replica.
By routing randomly to one of the replicas, there is a high chance that a client will reach a benign replica upon a retry.

I hope that answers your question!

5 Likes
3

Thank you for the clarification.

A couple of additional questions:

  • Would it make sense for the boundary nodes to choose a random replica among the nodes in its region (among the 4 it is present in) instead of a random replica from the 13 nodes in the subnet. That way we won’t be running into the worst case scenario of the boundary node making round trips from the node farthest away from it.
  • Suppose we decide to run our own boundary node, would it be possible for us to modify the routing behaviour to route requests to the closest replica instead of the default behaviour of routing to a random node by changing some configuration values instead of forking and maintaining a forked codebase?

The motivation behind this is to treat the subnet as one would treat their georeplicated database where read-only replicas serve requests from the closest region and writes are written to all write replicas that participate in consensus.

Thoughts?

1 Like
4

This is actually really interesting and the first time I heard about this. Thanks for clarifying @rbirkner

What could be possible ways around that in the future?

If it could stay under 100ms that would already solve for 80% of the use-cases. And if it stayed in the 10ms-50ms range it would even solve for modern gaming/streaming requirements.

I tried starting a discussion about this issue a while ago, but it didn’t really pick up:

Maybe because I had “proposal” in the title, which looking back is a bit misleading. I wanted to see the community discuss possible approaches and solutions to these issues.

I think this is important for many reasons:

  • SEO: search engines rate webpages based on loading speeds, and ping plays an important role in that. So as long as there isn’t a solution for this, I’m forced to keep the frontpages of apps, blogs, etc hosted on web2.
  • Streaming/voice: any form of streaming or voice-chat app needs decent ping, so randomly going between 30ms and 400ms doesn’t really work.
  • Gaming: same thing. Any ping over 100ms makes pretty much any modern game unplayable. So if Dfinity is serious about going into that market, serving based on geolocation seems like a must have.

Either way, interested in where this discussion goes. What is your use case @saikatdas0790? is this causing issues for hot or not? Do you plan to add streaming?

5

Thanks a lot for the discussion. @erikblues I completely agree that this is something we should discuss and think about different options (e.g., more aggressive caching could “mask” the high RTTs to go to a replica).

@saikatdas0790

Would it make sense for the boundary nodes to choose a random replica among the nodes in its region (among the 4 it is present in) instead of a random replica from the 13 nodes in the subnet.

Since up to 1/3 of the nodes could be malicious, routing only to a single region might lead to only “hitting” malicious nodes. Of course, this is an extreme case that might in reality never manifest itself.

Suppose we decide to run our own boundary node, would it be possible for us to modify the routing behaviour…

In theory, this is possible. However, at the moment, there is no possibility to run your own boundary node. Once we roll out the API boundary nodes, the NNS will deploy and control the API boundary nodes. Changes to the routing would be part of a normal release, which goes through a proposal and a vote on the NNS.

@erikblues
One thing that is important to remember is that all “state changes” go through consensus. In all these cases, the latency doesn’t matter that much as the time to go through consensus is in the order of 1-2s.

For read requests (e.g., video streaming), I think more aggressive caching will help a lot. What kind of requests did you have in mind?

2 Likes
6

Our use case is SSR edge rendering on the edge and expecting the edge node to hit the closest IC node to fetch data for lowest possible request latencies

1 Like