# Request for DFINITY Team Assistance in Investigating Potential Issues with the lhg73 Subnet **URL:** \ **Category:** Developers\ **Tags:** Discussing, nns\ **Created:** [February 28, 2026, 3:45pm UTC](https://forum.dfinity.org/t/request-for-dfinity-team-assistance-in-investigating-potential-issues-with-the-lhg73-subnet/64921 "2026-02-28T15:45:26Z")\ **Posts on this page:** 16\ **Page:** 2
**Author:** ![Lorimer](https://sea1.discourse-cdn.com/flex023/user_avatar/forum.dfinity.org/lorimer/32/17601_2.png) [@Lorimer](https://forum.dfinity.org/u/Lorimer)\ **Post date:** [February 28, 2026, 6:36pm UTC](https://forum.dfinity.org/t/request-for-dfinity-team-assistance-in-investigating-potential-issues-with-the-lhg73-subnet/64921/21 "2026-02-28T18:36:19Z")
Hotfix → > **[fix: More consistent \`remaining\_canisters\` · dfinity/ic@a62fbc6](https://github.com/dfinity/ic/commit/a62fbc672#diff-7a73591b22c69dc0e601cdac8d23228fbd5293a57a549a63f5e2bbcf9fbbda09L373-L522)** > > Use the same source of truth for the iterator and \`remaining\_canisters\`. ---
**Author:** ![Lorimer](https://sea1.discourse-cdn.com/flex023/user_avatar/forum.dfinity.org/lorimer/32/17601_2.png) [@Lorimer](https://forum.dfinity.org/u/Lorimer)\ **Post date:** [February 28, 2026, 6:51pm UTC](https://forum.dfinity.org/t/request-for-dfinity-team-assistance-in-investigating-potential-issues-with-the-lhg73-subnet/64921/22 "2026-02-28T18:51:36Z")
Looks like a divide by zero error was introduced due to `subnet_schedule.len()` differing to `state.canister_states().len()`, and the [division here](https://github.com/dfinity/ic/commit/a62fbc672#diff-7a73591b22c69dc0e601cdac8d23228fbd5293a57a549a63f5e2bbcf9fbbda09L521). I guess this recurs after a restart so the subnet stalls. I’m not clear why this would cause degraded performance on other subnets (rather than just fully fledged stalling), so it may actually be unrelated to what’s being experienced on this particular subnet (lhg73). I’ll avoid speculating further ---
**Author:** ![Zane](https://sea1.discourse-cdn.com/flex023/user_avatar/forum.dfinity.org/zane/32/6541_2.png) [@Zane](https://forum.dfinity.org/u/Zane)\ **Post date:** [February 28, 2026, 8:16pm UTC](https://forum.dfinity.org/t/request-for-dfinity-team-assistance-in-investigating-potential-issues-with-the-lhg73-subnet/64921/23 "2026-02-28T20:16:23Z")
I think [this](https://github.com/dfinity/ic/commit/a62fbc672#diff-7a73591b22c69dc0e601cdac8d23228fbd5293a57a549a63f5e2bbcf9fbbda09L521) change was mainly done to make the code more robust and cover some rare edge cases like newly created subnets with no canisters on them, that division has been there for many months and the `number_of_canisters` value hasn’t been changed, so it’s unlikely to have caused the downtimes. `subnet_schedule` and `canister_states` lengths diverging, with the former being greater, could have played a role. Generally they are kept in sync throughout the codebase, but here they could have different values cause `canister_priority_mut` is called for all canister ids that were executed in the round and it uses `subnet_schedule::get_mut`, which adds a default entry in the priorities map if one doesn’t exist for the provided id. So if a canister was deleted in the round but still registered in the executed set it would be added back to the subnet\_schedule. ---
**Author:** ![let4be](https://sea1.discourse-cdn.com/flex023/user_avatar/forum.dfinity.org/let4be/32/25128_2.png) [@let4be](https://forum.dfinity.org/u/let4be)\ **Post date:** [February 28, 2026, 9:09pm UTC](https://forum.dfinity.org/t/request-for-dfinity-team-assistance-in-investigating-potential-issues-with-the-lhg73-subnet/64921/24 "2026-02-28T21:09:50Z")
so, what’s up with lhg73? still broken, been hours ---
**Author:** ![icprobot](https://sea1.discourse-cdn.com/flex023/user_avatar/forum.dfinity.org/icprobot/32/18752_2.png) [@icprobot](https://forum.dfinity.org/u/icprobot)\ **Post date:** [March 1, 2026, 12:19am UTC](https://forum.dfinity.org/t/request-for-dfinity-team-assistance-in-investigating-potential-issues-with-the-lhg73-subnet/64921/25 "2026-03-01T00:19:37Z")
I can’t trade either, what’s going on ---
**Author:** ![ICPSwap](https://sea1.discourse-cdn.com/flex023/user_avatar/forum.dfinity.org/icpswap/32/11193_2.png) [@ICPSwap](https://forum.dfinity.org/u/ICPSwap)\ **Post date:** [March 1, 2026, 1:31am UTC](https://forum.dfinity.org/t/request-for-dfinity-team-assistance-in-investigating-potential-issues-with-the-lhg73-subnet/64921/26 "2026-03-01T01:31:24Z")
Could any member of the DFINITY team please help notify the relevant devs? The lhg73 subnet appears to be down — it has been showing 0 Cycles/s for quite some time. ![image](https://us1.discourse-cdn.com/flex023/uploads/dfn/original/3X/4/8/48942fa758cf153e94f738b5a9bd050be707289c.png) [https://dashboard.internetcomputer.org/network/subnets/lhg73-sax6z-2zank-6oer2-575lz-zgbxx-ptudx-5korm-fy7we-kh4hl-pqe](https://dashboard.internetcomputer.org/network/subnets/lhg73-sax6z-2zank-6oer2-575lz-zgbxx-ptudx-5korm-fy7we-kh4hl-pqe) ---
**Author:** ![Arthur2](https://avatars.discourse-cdn.com/v4/letter/a/b487fb/32.png) [@Arthur2](https://forum.dfinity.org/u/Arthur2)\ **Post date:** [March 1, 2026, 5:44am UTC](https://forum.dfinity.org/t/request-for-dfinity-team-assistance-in-investigating-potential-issues-with-the-lhg73-subnet/64921/27 "2026-03-01T05:44:52Z")
Bump [@DFINITYTeamMembers](https://forum.dfinity.org/groups/dfinityteammembers) @andrewbattat ---
**Author:** ![Badri](https://sea1.discourse-cdn.com/flex023/user_avatar/forum.dfinity.org/badri/32/35363_2.png) [@Badri](https://forum.dfinity.org/u/Badri)\ **Post date:** [March 1, 2026, 8:10am UTC](https://forum.dfinity.org/t/request-for-dfinity-team-assistance-in-investigating-potential-issues-with-the-lhg73-subnet/64921/28 "2026-03-01T08:10:33Z")
Bump [@DFINITYTeamMembers](https://forum.dfinity.org/groups/dfinityteammembers) ---
**Author:** ![let4be](https://sea1.discourse-cdn.com/flex023/user_avatar/forum.dfinity.org/let4be/32/25128_2.png) [@let4be](https://forum.dfinity.org/u/let4be)\ **Post date:** [March 1, 2026, 8:35am UTC](https://forum.dfinity.org/t/request-for-dfinity-team-assistance-in-investigating-potential-issues-with-the-lhg73-subnet/64921/29 "2026-03-01T08:35:09Z")
Does anyone care to explain why this 16+ hours downtime is NOT mentioned on the [https://status.internetcomputer.org](https://status.internetcomputer.org) ? ---
**Author:** ![let4be](https://sea1.discourse-cdn.com/flex023/user_avatar/forum.dfinity.org/let4be/32/25128_2.png) [@let4be](https://forum.dfinity.org/u/let4be)\ **Post date:** [March 1, 2026, 8:35am UTC](https://forum.dfinity.org/t/request-for-dfinity-team-assistance-in-investigating-potential-issues-with-the-lhg73-subnet/64921/30 "2026-03-01T08:35:47Z")
Clearly, NOT all systems are operational ![image](https://us1.discourse-cdn.com/flex023/uploads/dfn/original/3X/1/0/10d8030ea65b18ac6a3a87e6cd016c3ebfc18b6a.png) ---
**Author:** ![Henn91](https://avatars.discourse-cdn.com/v4/letter/h/edb3f5/32.png) [@Henn91](https://forum.dfinity.org/u/Henn91)\ **Post date:** [March 1, 2026, 10:01am UTC](https://forum.dfinity.org/t/request-for-dfinity-team-assistance-in-investigating-potential-issues-with-the-lhg73-subnet/64921/31 "2026-03-01T10:01:26Z")
lhg73-sax6z-2zank-6oer2-575lz-zgbxx-ptudx-5korm-fy7we-kh4hl-pqe Authorized only, shouldnt it be Public subnet? ---
**Author:** ![Lorimer](https://sea1.discourse-cdn.com/flex023/user_avatar/forum.dfinity.org/lorimer/32/17601_2.png) [@Lorimer](https://forum.dfinity.org/u/Lorimer)\ **Post date:** [March 1, 2026, 10:08am UTC](https://forum.dfinity.org/t/request-for-dfinity-team-assistance-in-investigating-potential-issues-with-the-lhg73-subnet/64921/32 "2026-03-01T10:08:54Z")
Historically it became private because the memory usage on that subnet surpassed certain thresholds. > [@On another topic](https://forum.dfinity.org/t/37965/1): > > lhg73-sax6z-2zank-6oer2-575lz-zgbxx-ptudx-5korm-fy7we-kh4hl-pqe Application false Subnet has more than 300 GiB state size [Those thresholds are now significantly higher](https://forum.dfinity.org/t/1-mission70-repricing-replicated-subnet-memory/64648/76), so based on that metric this subnet could be made public today. cc @alexu ---
**Author:** ![ZackDS](https://sea1.discourse-cdn.com/flex023/user_avatar/forum.dfinity.org/zackds/32/40368_2.png) [@ZackDS](https://forum.dfinity.org/u/ZackDS)\ **Post date:** [March 1, 2026, 10:35am UTC](https://forum.dfinity.org/t/request-for-dfinity-team-assistance-in-investigating-potential-issues-with-the-lhg73-subnet/64921/33 "2026-03-01T10:35:49Z")
Not sure what the current issue is that apparently only affects some of the 25K canisters, there were a few deployed on 27th and there was some testing or smth as shown by the roughly 14 GB temporary state increase but even that dropped back. ![image](https://us1.discourse-cdn.com/flex023/uploads/dfn/original/3X/3/2/3276a7a7510d202f040b748abdd4d0b9cab75c55.png) Nodes seem to be healthy, only thing is to wait for the hotfixed replica to be deployed I guess ? ![Screenshot 2026-03-01 121927](https://us1.discourse-cdn.com/flex023/uploads/dfn/original/3X/1/4/14578e0cfd8375c613a3b07ad37fba6c4ea06d21.png) ---
**Author:** ![let4be](https://sea1.discourse-cdn.com/flex023/user_avatar/forum.dfinity.org/let4be/32/25128_2.png) [@let4be](https://forum.dfinity.org/u/let4be)\ **Post date:** [March 1, 2026, 11:48am UTC](https://forum.dfinity.org/t/request-for-dfinity-team-assistance-in-investigating-potential-issues-with-the-lhg73-subnet/64921/34 "2026-03-01T11:48:21Z")
zzz, any update? been 24h since subnet is down ---
**Author:** ![ICPSwap](https://sea1.discourse-cdn.com/flex023/user_avatar/forum.dfinity.org/icpswap/32/11193_2.png) [@ICPSwap](https://forum.dfinity.org/u/ICPSwap)\ **Post date:** [March 1, 2026, 12:14pm UTC](https://forum.dfinity.org/t/request-for-dfinity-team-assistance-in-investigating-potential-issues-with-the-lhg73-subnet/64921/35 "2026-03-01T12:14:14Z")
Update: Several devs have responded to us on Slack. With their investigation and support, our dev team expects to fully restore normal trading soon. ---
**Author:** ![stefan.schneider](https://avatars.discourse-cdn.com/v4/letter/s/a3d4f5/32.png) [@stefan.schneider](https://forum.dfinity.org/u/stefan.schneider)\ **Post date:** [March 1, 2026, 12:16pm UTC](https://forum.dfinity.org/t/request-for-dfinity-team-assistance-in-investigating-potential-issues-with-the-lhg73-subnet/64921/36 "2026-03-01T12:16:39Z")
We are in contact with the ICPSwap team helping them debugging their canister. The subnet is not down. Let me give some more context on the subnet metrics you are observing. The increased state size during the stall of the `lspz2` subnet yesterday, the increased cycle burn during the same time period and the (supposed) 0 cycle burn since then are all the the same effect. When a canister does a XNet call, both memory and cycles are reserved while waiting for the response. Yesterday, any calls to `lspz2` were stuck. During that time, both the cycles and the memory were therefore withheld, and only refunded when the subnet got unstuck. As soon as `lspz2` was back online, the memory was refunded, as you can see in the graphs that @ZackDS shared. For the cycles, an unfortunate quirk of the public dashboard is that it does not show cycle refunds as negative cycle burn. Instead, it shows 0 cycle burn until the refunded amount is properly burned. It’s more obvious what is happening from the internal dashboards: ![Screenshot 2026-03-01 at 10.53.39](https://us1.discourse-cdn.com/flex023/uploads/dfn/original/3X/4/5/45eb0a7e439c8d236007e5017227eb7815e3e5ad.png) As you can see here, the cycles consumed sharply drops (due to the refunds) and the subnet has been burning more cycles ever since. Unfortunately the public dashboard will show 0 burn until the total cycles consumed exceeds the historic peak again. But crucially, that does not mean that the subnet is down. It is however true that ICPSwap did not fully recover after the other subnet came back online, which is likely an effect of the uncommon errors their canisters received during the stall of the lspz2 subnet. We are trying to help the ICPSwap team to understand what exactly happened to their canisters and how to fix it. [Previous page](https://forum.dfinity.org/t/request-for-dfinity-team-assistance-in-investigating-potential-issues-with-the-lhg73-subnet/64921.md?page=1)