Just curious: do replicas perform any rate-limiting? Boundary nodes only handle ingress messages, but what if a single canister makes an excessive number of inter-canister calls within a subnet?
Hey, flow control on the Internet computer is an involved topic. I would try and be brief here but this is a good candidate for a blog entry.
Replica and the networking connecting the replicas operate at disparate processing speeds. At any given time some replicas may be faster/slower than the others (asynchronous nature of the network). IC needs to employ intelligent queuing/buffering to keep such nodes in sync in spite of the asynchronous nature of the underlying network. Flow control for the IC broadly has to take care of.
- Ingress message
- Consensus messages.
- Cross canister message part of the execution.
Ingress message flow control: Leaky bucket, where messages that don’t fit in the queue are dropped. This is fine as the user agent can retry the message. Fairness and thus no starvation are goals here. Once an ingress makes it into the replicas ingress pool, it can still be dropped as it can expire before it is picked up by the block maker for consensus/execution. This is the only layer where we can tolerate message drops.
Consensus: The P2P layer implements fixed-sized i/o queues per replica. So malicious replicas cannot overwhelm other replicas. These queues are sized (size and count of messages) sufficiently well such that replicas operating within the protocol limits will never run out of queue space. I a replica genuinely falls behind (transient sluggish network) it can ask for retransmission to do a quick resync. If a part of the subnet is lagging behind for longer periods of time - it will fail to do quick resyncs and eventually fall back to CUP-based state-syncs. This queuing model works on the premise the CUP-based state syncs are orders of magnitude faster than actual consensus-based progress.
Message routing: Flow control in message routing is similar to P2P routing in essence. XNet messaging employs Stream Transmission Protocol (look up this description by David Derler). It’s very close to TCP ordered message delivery. Again Per canister per subnet fixed-size, message routing queues make sure that no single canister/subnet is overwhelming the IC network bandwidth. The goals in messaging routing are: Guaranteed, ordered delivery of messages across canisters. Since there is the promise of guaranteed delivery - something has to give in if the processing slows down. So if a canister makes exorbitantly high cross net messages its execution may slow down as queues back up.
This is a high-level description. Flow control is a well-researched academic topic, and the IC overall does a great job at it (wrtx to another blockchain tech out there.) Cross net messaging and subnets are one of the key innovations of the IC.