Issue on subnet jtdsg-3h6gi-hs7o5-z2soi-43w3z-soyl3-ajnp3-ekni5-sw553-5kw67-nqe

General
1

Hello,
There seems to be something odd going on with the subnet : jtdsg-3h6gi-hs7o5-z2soi-43w3z-soyl3-ajnp3-ekni5-sw553-5kw67-nqe

The number of block per seconds has been dropping since 8AM UTC time.

Many of my canisters are deployed in this subnet and since this morning when I run into issues when I try to upgrade.

Error: The Replica returned an error: code 5, message: "Canister p4y2d-yyaaa-aaaaj-qaixa-cai trapped explicitly: canister_pre_upgrade attempted with outstanding message callbacks (try stopping the canister before upgrade)"

Upgrading is not the thing only causing issues, update calls and inter-canister are taking way more time than expected. (For instance : asking for the canister balance trough inter-canister call to the NNS ledger taking 1min or timing out).
Query calls seem to be processed in an expected timing.

A few questions :

What is happening there ?
Is there a currently a way to move a canister from one subnet to another (or is it something planned) ?
Is there a way to deploy into one specific subnet, if not who/what is taking the decision?

2 Likes
2

Is there any way to check , in which subnet your canister is deployed ? Because ic.rocks is not stable …
Also I am facing the same issue thats why i asked this to you.

3

You can go to the dashboard : https://dashboard.internetcomputer.org/ and search for your canister, it will automatically detect in which subnet you are.

The issue seems to be resolved for me. If someone from Dfinity check this topic I would be curious to know what happened.

4

Thanks for the ping @Seb, we’ll look into it

1 Like
5

jtdsg experienced high load yesterday and that activated our safeguard mechanism that protects against replicas going out-of-memory by delaying execution of messages. That caused increased latencies for some messages.

@free found one optimization for the safeguard mechanism that would drastically reduce the latency under load. We’ll try to implement that soon.

6 Likes