Postmortem: lhg73 subnet incident on Dec 26th 2024

[Lorimer]

For anyone who’s interested →

Post-Mortem Update: Review of the SSH Accessibility after the `lhg73` Subnet Recovery

Hello ICP Community, Following the recent lhg73 subnet incident, our team conducted a post-mortem analysis to identify and address the root causes that impeded the recovery process. One action item was to understand why DFINITY engineers were unable to SSH into the lhg73 nodes during recovery, even after the adoption of Proposal 134623. As part of the subnet recovery process, DFINITY engineers SSH as read-only users into all subnet nodes to compare states between nodes and ensure that the recov…