Subnet lhg73 is stalled

[sat]

The issue was already mentioned in another forum thread.

Subnet 'lhg73' MIEPs Suddenly Dropped to 0 - Request for Guidance

Dear everyone, The MIEPs for the subnet ‘lhg73’, where ICPSwap DApp is located, have suddenly dropped to 0. As a result, the ICPSwap DApp is currently inaccessible. While we await investigation and updates subnet from the DFINITY technical team, is there anything else we can do on our end? Does anyone have any insights or suggestions? Thank you very much for your guidance and support! P.S. The backup version, or test environment of the ICPSwap DApp, at https://test.app.icpswap.com/swap, is fu…

Seems like this is a very unlikely corner case in the consensus code. We will submit an NNS proposal to recover the subnet, and continue debugging the root cause with the data after the incident is resolved.

[sat]

So we just ran a simple subnet recovery with the following proposals:

• halt the subnet: https://dashboard.internetcomputer.org/proposal/134605
• prepare the catch-up package: https://dashboard.internetcomputer.org/proposal/134606 (which took some time due to the size of the subnet)
• unhalt the subnet: https://dashboard.internetcomputer.org/proposal/134607

This recovered the subnet, although we did not resolve the root cause. The consensus team will try to get to the bottom of it, although they only have limited resources over the holidays.
There will also likely be a Post Mortem, once we identify and resolve the root cause.

[timk11]

Voted to adopt proposals 134605, 134606 and 134607. Critical fix as explained above, already executed. Great work @sat and team on getting this happening so fast!

[Lorimer]

Subnet Management - lhg73 (Application)

Proposal 134605 TLDR: I’ve voted to adopt. Halt subnet lhg73-sax6z-2zank-6oer2-575lz-zgbxx-ptudx-5korm-fy7we-kh4hl-pqe, for recovery and update ssh readonly access This proposal has already executed, at 2024-12-26, 19:00:56 UTC (minutes after it was submitted several hours ago). As can be seen in the screenshot below, the subnet had stalled and was no longer processing blocks for a couple of hours prior to this proposal to halt the subnet. [image] . It’s worth noting that this…

Subnet Management - lhg73 (Application)

Proposal 134606 TLDR : I’ve voted to reject Update recovery CatchUp Package of subnet: lhg73 to height: 130666000 I voted to reject this CUP proposal, the reasoning for which I’ve previously discussed with DFINITY See the prior proposal above for context .

Subnet Management - lhg73 (Application)

Proposal 134607 TLDR: I’ve voted to adopt. Unhalt subnet lhg73-sax6z-2zank-6oer2-575lz-zgbxx-ptudx-5korm-fy7we-kh4hl-pqe, for recovery and update ssh readonly access This proposal has already executed, at 2024-12-26, 20:43:06 UTC (minutes after it was submitted several hours ago). As can be seen in the screenshot below, after stalling, halting, CUP, and unhalting, the subnet is back on track and processing blocks (with the condition improving since the unhalt proposal executed). …

[sat]

Here is a follow-up (related) forum thread:

Security hotfix ongoing

We have identified a security issue and are following the procedure described in NNS proposal 48792: Security Patch Policy and Procedure. An NNS proposal containing the fix will be submitted shortly. More information about what is being released will be posted here after the rollout of the security hotfix completed.

[Mar]

It seems to be stalled again.

[ICPSwap]

Subnet ‘lhg73’ stalled again within 24 hours. Any insights on this?

[sat]

Yes, we recovered the subnet yesterday without fixing the root cause, and just when the fix for the root cause was about to be rolled out to the subnet, it got stuck again. Bad luck.

[sat]

We started the recovery process again. Recovery should be faster this time.

[sat]

The lhg73 subnet is back up, and running the version with the fix for the root cause. We’ll proceed with the rollout of the fix to the other subnets.

[Mar]

For what it’s worth, this subnet is unusable for us, latency is too high and unpredictable for a social network. As a result, we will be moving to another one, with all the inconveniences that it creates. I’m not sure if this is the expected behavior.

[timk11]

Voted to adopt proposals 134623, 134629 and 134632. Critical hotfix for subnet lhg73 having stalled again, already executed pending further investigation into cause as explained above.

[Lorimer]

Subnet Management - lhg73 (Application)

Proposal 134623 TLDR: I’ve voted to adopt. Halt subnet lhg73-sax6z-2zank-6oer2-575lz-zgbxx-ptudx-5korm-fy7we-kh4hl-pqe, for recovery and update ssh readonly access This proposal has already executed, at 2024-12-27, 18:09:58 UTC (minutes after it was submitted hours ago). As can be seen in the screenshot below, the subnet had stalled and was no longer processing blocks for a handful of minutes prior to this proposal to halt the subnet. [image] This is the third recent stall, w…

Subnet Management - lhg73 (Application)

Proposal 134629 TLDR : I’ve voted to reject Update recovery CatchUp Package of subnet: lhg73 to height: 130812000 I voted to reject this CUP proposal, the reasoning for which I’ve previously discussed with DFINITY See the prior proposal above for context

Subnet Management - lhg73 (Application)

Proposal 134632 TLDR: I’ve voted to adopt. Unhalt subnet lhg73-sax6z-2zank-6oer2-575lz-zgbxx-ptudx-5korm-fy7we-kh4hl-pqe, after recovery and update ssh readonly access This proposal has already executed, at 2024-12-27, 19:12:34 UTC (minutes after it was submitted several hours ago). As can be seen in the screenshot below, after stalling, halting, CUP, and unhalting, the subnet is back on track and processing blocks (with the condition improving since the unhalt proposal executed). …

[LaCosta]

Voted to adopt proposals 134623, 134629 and 134632.
This subnet stalled not long ago and was recovered without fixing the root cause and just before this was fixed it stalled again. The issue was discussed here and the following IC OS Version Election proposals 134608 and 134609 were made and have been executed to the issue should be solved. Great work @sat and team.