Path forward for subnet splitting and protocol scaling

[oggy]

@levi

There is a limited number of messages that can execute within one round so a canister call to another canister on the same subnet can still unpredictably require a consensus and networking round in between the two canisters’ message-executions.

Yes this is the case currently (and probably will be to some extent for the foreseeable future), but I don’t think that anyone really wants unpredictable execution latency and throughput as the norm? We won’t get to a predictable environment within a few months for sure, but we can improve gradually.

@levi

Another factor here is that the end-to-end ingress message end-user latency difference between calling a canister that calls another canister on the same subnet and calling a canister that calls another canister on a different subnet is much less of a difference (maybe around 3x max?).

IIRC, with the sync call endpoint, an ingress call takes around 1.2 seconds to get a response, and the median cross-subnet call is around 6 seconds (though this is largely due to the NNS having a slower finalization rate - otherwise it is indeed probably around 3 seconds). But do note that this gets worse if there are multiple calls being made. Are people OK with an 3x increase? Famously Amazon found that they’d lose 1% of income with every 100ms of latency added, but I can accept that maybe it’s not so bad for everyone.

@levi

For the plan in the original post, we can say that every canister call takes up to the time of a cross-subnet call and sometimes it can be faster if the protocol is able to make it faster (if both canisters happen to be on the same subnet at the time of the call) but canister authors should have in mind that a canister call can always take up to the time of a cross subnet call.

This is how the original philosophy behind the system. You couldn’t choose the subnet that you deploy canisters on. In principle, creating a canister from a canister could’ve created that canister anywhere, not just on the same subnet - but it didn’t in the implementation. But people have regardless noticed that there was a performance difference between local and cross-net calls and have come to rely on that, and I think that’d be quite difficult to change now. I mean one could of course now make local calls slower on purpose so that they don’t come to rely on the performance, but think that that would leave a lot of people unhappy. Similar with composite queries.

@levi

The cycles cost of a canister call can be made the same whether or not the canisters are on the same subnet, the cost can be set by looking at the average resource-usage of the canister calls throughout the whole network.

This is already the case today - every inter-canister call costs the same.