Path forward for subnet splitting and protocol scaling

[levi]

@oggy

@levi

Another factor here is that the end-to-end ingress message end-user latency difference between calling a canister that calls another canister on the same subnet and calling a canister that calls another canister on a different subnet is much less of a difference (maybe around 3x max?).

IIRC, with the sync call endpoint, an ingress call takes around 1.2 seconds to get a response, and the median cross-subnet call is around 6 seconds (though this is largely due to the NNS having a slower finalization rate - otherwise it is indeed probably around 3 seconds).

Great, less than 2.5x. Looks like the recent reduction of notarization delays is working. And this is 2.5x in the context of the difference between 1.2 seconds and 3 seconds. The plan in the original post does not affect calls to the NNS subnet since they are already cross-subnet calls.

@oggy

But do note that this gets worse if there are multiple calls being made.

If there are multiple calls being made it gets worse for both same-subnet and cross-subnet calls so it doesn’t make a difference to the plan in the original post.

@oggy

Are people OK with an 3x increase?

Let’s be clear on where the increase will happen. It will only increase from 1.2 seconds to 3 seconds for canister-methods that currently call other canisters on the same subnet. Canister-methods that currently call canisters on other subnets, and canister-methods that don’t make any downstream calls, will not be affected in any way by the plan in the original post. Most canister calls on the network are already cross-subnet, every canister call to the ICP or SNS ledgers are already cross-subnet and will not be affected in any way by this plan.

@oggy

Famously Amazon found that they’d lose 1% of income with every 100ms of latency added, but I can accept that maybe it’s not so bad for everyone.

The internet-computer’s values of decentralization, security, replication, infinite-scalability, availability/liveness, and consensus, are worth more than millisecond optimizations in my view.

@oggy

but I don’t think that anyone really wants unpredictable execution latency and throughput as the norm?

Cross-canister calls taking between 1.2 and 3.0 seconds is a predictable latency. It is much more predictable than what we have now, where anyone can install thousands of canisters on a subnet and run heartbeats on all of them and then the other canisters on that subnet can take minutes/hours to complete a cross-canister call.

In the plan in the original post, even if someone installs a million canisters on a single subnet, every canister in that subnet will maintain the consistent latency of up to 3 seconds max, since the protocol will just keep splitting the subnet till the load is balanced.

The protocol must breath in and out for the balance.

@oggy

@levi

For the plan in the original post, we can say that every canister call takes up to the time of a cross-subnet call and sometimes it can be faster if the protocol is able to make it faster (if both canisters happen to be on the same subnet at the time of the call) but canister authors should have in mind that a canister call can always take up to the time of a cross subnet call.

This is how the original philosophy behind the system. You couldn’t choose the subnet that you deploy canisters on. In principle, creating a canister from a canister could’ve created that canister anywhere, not just on the same subnet - but it didn’t in the implementation.

Thanks for sharing that is cool to know, that does align with the infinite scalability plans.

@oggy

But people have regardless noticed that there was a performance difference between local and cross-net calls and have come to rely on that, and I think that’d be quite difficult to change now.

I don’t see how it is possible that people have come to “rely” on that unless it is a rented subnet. Anyone can install canisters on a public subnet and run heartbeats or any computation, and then all other canisters on the subnet can take much longer to complete a same-subnet canister call, since all the canisters on one subnet share the same block space.

Another factor is that few weeks ago before the reduction in notarization delays, the time it used to take to do a same-subnet call is now the same as the time it takes to do a cross-subnet call.

@oggy

I mean one could of course now make local calls slower on purpose so that they don’t come to rely on the performance, but think that that would leave a lot of people unhappy.

No need to do that. Since same-subnet canister-calls on public subnets can already take much longer than usual when there is load on the subnet, it would be incorrect even now for canister authors to rely on any specific or consistent latency for the correctness of the code.

@oggy

This is already the case today - every inter-canister call costs the same .

Thanks, my mistake. That makes this plan even smoother.