The data that made it onto the replicas via these side-channels would still be technically part of the block. A block with only a hash and no actual data would be more or less useless. So for the purposes of gossiping the block around and achieving consensus, the block would still have a 4 MB limit. But for the upper layers (or if you wanted to back up and preserve the full blockchain) you would have to include these additional pieces of content.
The reason why there’s a relatively minuscule limit on block size is that the block must be gossiped to all (well, most) replicas on the subnet in order to achieve consensus. And eventually to all of them in order to be executed. That means reliably sending the full block across the Atlantic and Pacific in well under one second. But if there was some way to know that all (or most) replicas already have some piece of content, all they would have to agree on is its “content ID” (i.e. hash). So there would be less data to gossip in this tiny time window. And the actual payload could have taken arbitrarily long to be uploaded to or generated by every replica independently, without holding back subnet progress.
So I fully agree with your concern and this would definitely not change anything in terms of safety / tamper resistance / trust.