Postmortem: lhg73 subnet incident on Dec 26th 2024

[icpalist]

@Kamil

There are no alerts for invalid consensus artifacts - when a replica deems a consensus artifact it received from a peer invalid it removes it from its pool, emits a warning message, and increments a metric: consensus_invalidated_artifacts. A replica should receive an invalid artifact only in two cases: either there is a bug in implementation which causes a peer to produce an invalid artifact or the peer acts maliciously. Currently there are no consumers of consensus_invalidated_artifacts and no alert is emitted when the counter increments. Having an alert would help to spot such issues earlier.

So if nodes in a subnet collude (see https://forum.dfinity.org/t/sybiling-nodes-exploiting-ic-network-community-attention-required) not even Dfinity will know.

Imagine IC having TVL of billions in DeFi apps. It would be too tempting to nodes to not try, having no stake to lose.