Request for DFINITY Team Assistance in Investigating Potential Issues with the lhg73 Subnet

Developers
,
1

We would greatly appreciate it if the DFINITY team could assist in investigating whether there may be an issue with the lhg73 subnet. Thank you very much in advance for your support. @sat

Currently, most trading pairs on ICPSwap are experiencing swap failures. Based on our preliminary investigation, trading pairs recently deployed on the pzp6e subnet appear to be functioning normally.

After experiencing three previous outages on the lhg73 subnet, we decided to deploy newly created trading pair canisters to the pzp6e subnet instead. For example, the TENDY (SNS)/ICP trading pair is operating normally.

However, trading pair canisters that were previously deployed on the lhg73 subnet are now encountering errors, such as the ICS/ICP trading pair.

We have also shared this information in Slack and posted it on the forum. Given the urgency of the situation, we kindly request that the relevant members of the DFINITY team help investigate this matter as soon as possible.

Thank you again for your continued support and assistance.

image
image1236×762 110 KB

image
image1749×963 94.1 KB

6 Likes
2

Bump @DFINITYTeamMembers

1 Like
3

Tokens on Oisy has serious issues too, custom tokens.

Loading token: The token with Ledger canister s2fo3-syaaa-aaaam-aftkq-cai could not be loaded because it’s currently not responding.

Adding custom token: The specified address is not a valid token in the selected network.
Something went wrong while validating the Ledger canister. / Invalid request expiry: Specified ingress_expiry not within expected range: Minimum allowed expiry: 2026-02-28 15:42:59.619418107 UTC, Maximum allowed expiry: 2026-02-28 15:48:29.619418107 UTC, Provided expiry: 2026-02-28 07:14:00 UTC. Provided ingress expiry time is 5 minutes.

Saving hidden token to wallet list: Something went wrong while saving the token. / The replica returned a rejection error: Request ID: 2d0058b541605fce702c866959f2ceebf842958b725cf59f65d9dce461b14e01 Reject code: 5 Reject text: Error from Canister doked-biaaa-aaaar-qag2a-cai: Canister called ic0.trap with message: 'Version mismatch, token update not allowed.

4

Dear Henn,

Thank you for the information.

May I ask whether you have tried restarting your device or synchronizing your system time? If the issue still persists, I am not sure whether it could be related to the recent incident affecting the lspz2 subnet:

5

Yes i synchronizing my system time, it didnt fix anything. Issue is wide area, not just time problem.

1 Like
6

Any word from dfinity?

1 Like
7

Yea, what’s up? getting some ETA would be nice…

1 Like
8

https://forum.dfinity.org/t/subnet-management-lspz2-application/35322/89

The subnet was halted earlier today and has only recently had a proposal to unhalt the subnet.

2 Likes
9

Actually, the latest proposal hasn’t set is_halted=false, it’s just set the recovery CUP. So I guess we’ll have another one incoming that actually unhalts the subnet so I can pick up from the CUP.

Proposal: 140604 - ICP Dashboard

1 Like
10

lhg73 is also cooked

1 Like
11

Give it to me in plainspeak. How much time do we have? Roughly

12

I’m not sure, I’m still trying to get a handle on what’s happening. I realised I confused two subnets above (lhg73 and lspz2).

The nodes are largely healthy, so I suspect it was a bad ICOS deployment. I’m currently looking at validating the CUP though

3 Likes
13

Someone broke it, Subnet replica time desynchronization (most likely)? If enough replicas on a subnet have incorrect system time or recover from downtime incorrectly, the subnet’s consensus time can jump.

1
1967×504 36.7 KB

2
2959×513 18.9 KB

3
3726×306 13.4 KB

2 Likes
14

Nice, no more errors in Oisy and trade with ICPSwap worked well.

15

This subnet has been running on ICOS version 007c473 for the last 3 or so days (since Proposal: 140547). This is the same ICOS version that many other subnets are running on, and was also running on Subnet lspz2 when it stalled earlier today (and has only recently been reverted to the prior ICOS version and had a CUP applied).

By the looks of things it would be worth rolling back the 007c473 ICOS deployment on more subnets than just lspz2.

cc @alexu

5 Likes
16

Dear Lorimer,
Thanks so much for the quick and thorough investigation

1 Like
17

Some pools from lhg73 are working again on icpswap, but ICP/ckUSDC still appears to be stuck(same subnet)

18

I tried again, and some swaps are now working normally.

I’m a bit curious — could this be related to the following incident?

Application Subnet is stalled
Incident status: Monitoring
A fix has been implemented and we are monitoring the results.
Time posted: Feb 28, 18:26 UTC

2 Likes
19

https://forum.dfinity.org/t/subnet-management-lspz2-application/35322/96

There is an identified issue in the ICOS version. I think we’ll need to stay tuned for more ICOS deployment proposals

2 Likes
20

a bit weird that some pools from the same subnet are now working again and some do not

1 Like