In talking to someone recently, it was mentioned that data throughput between subnets to any specific subnet is capped at 2-4 MB per block. So for example, all messages that hit the ICP Ledger on the NNS subnet are capped by the NNS subnet’s block rate of ~0.8 blocks/sec, equating to 0.8 blocks/sec * 2-4 MB/block = 1.6-3.2 MB potential egress per second.
A few follow-up questions to better understand:
Is this above characterization of cross subnet call data throughput limits to canister(s) on a specific subnet accurate?
What are the current limits and goals for of cross-subnet data ingress/egress?
What are some strategies that developers can employ to improve their throughput to a canister if they aren’t able to directly deploy a canister to that subnet (i.e. NNS/SNS subnet)?
Is there a current maximum throughput of data that can be transported between subnets per time interval (i.e. second), regardless of the number of subnets on the network?
If I understand correctly it is asymmetrical. Sending data to other subnets has massive bandwidth because a hash is sent and the remote subnet pulls it P2P. For each subnet individually, that subnet needs to get those files, so it is limited by what/when/how long it takes to pull.
So you can push far more out than you can take in(assuming you are communicating to multiple subnets).
This was my understanding, but I’m sure there is a much more technical explanation someone from DFINITY can give.
Austin is right. A subnet’s output (whether in the form of responses to ingress messages or canister to canister messages) is mostly limited by compute (IIRC you can only output a couple of GB per round before you run out of instructions).
You are also right about the block size. It is currently limited to 4 MB and this is because before you even get to 4 MB you run into limitations with the consensus algorithm and the speed of light: you simply cannot transmit a lot more than that around the world over the public internet with sub-second roundtrip latency (particularly considering that you need another couple of network roundtrips per consensus round). Meaning that a subnet’s ingress (not egress) including ingress messages, canister-to-canister messages, HTTP outcalls is limited to 4 MB.
The Consensus team is actively working on only putting references to messages into blocks, so that (i) block size and latency is reduced and (ii) the referenced payloads can be arbitrarily large. For now, they’re focusing on (i) (and only on ingress messages) with payload size still limited to 4 MB per block, only the block size and latency significantly reduced. Once they iron out all the rough edges, they will start looking into increasing the payload size; and applying the same approach to other payload types (XNet messages, HTTP outcalls).
To answer your last couple of questions, you can max out your throughput the usual way: make concurrent calls instead of waiting for a roundtrip to complete; this happens implicitly if your calls are each triggered by different incoming calls; otherwise you need to issue the calls all at once and join await all of them.
The current maximum throughput is 4 MB times the number of subnets. But there is no other limit beyond a subnet’s ingress bandwidth and the internet’s / the specific data centers’ network bandwidth.
Since you brought up the NNS, note that for the NNS in particular, there’s also another limit in place as a safeguard, which is that it will rate-limit all other subnets to ~100 messages per subnet per one block. So even if the messages are small, say ~500 bytes, you don’t get to send 2000 of them to the NNS in each round. IIRC this was largely a precaution to thwart DoS attacks on the NNS through cross-subnet calls (i.e., to ensure that there’s always enough bandwidth for ingress messages).
Got it, so this is a message throughput limit but not a message size limit.
If the messages are large, then am I correct in stating that the size of the message payload won’t matter (as much) anymore toward blocks throughput in terms of filling up a block once the Hashed block payloads feature that @free mentioned is implemented?
Edit: After reading a bit more it seems like this is mostly for ingress messages to start, and Xnet (inter-canister/subnet) messages are still a ways away, so in this case the size of the message payload still makes a difference in the produced subnet block size.