Frontend canister deployment failing for ic mainnet, timing out

I am facing a strange problem trying to deploy a frontend canister live on IC, been stuck for 2 days now thinking maybe it’s my internet connection or environment but now I think it’s something else on IC, I have main frontend with a few images, it’s giving this error

Caused by: Failed to create batch: The replica returned a rejection error: reject code SysTransient, reject message Ingress message 0x5c1f59d0702efd0187ad97d9a5f9c8824a9edd6e10a82626543e78b759782bae timed out waiting to start executing., error code None

I tried deploying it in on different computers, different wifi networks and even in a different city, locally it’s deploying in seconds, and on playground it’s also deploying in seconds, it got very strange when I tried to deploy a simple default hello frontend and it’s timing out again on mainnet while it’s deploying in seconds on local and playground.
What could this be?
Thanks!

1 Like

Maybe you should try using a proxy

I have the same issue. But on investigating further, it seems like all update calls to my wallet canister do fail (I’ve tried to transfer cycles from the wallet to the cycle ledger, and the call also fails with timeout error). Probably, one of the subnets is down? My wallet canister principal is qe56m-saaaa-aaaag-aacda-cai if it helps.

Do you know what subnet you are on?

Also @MaximGritsenko

You can run dfx --verbose deploy --ic to find out.

It seems that the problem was solved somehow. Today the requests to the wallets do pass. But in case there’s still investigation to be done, according to https://dashboard.internetcomputer.org/canister/qe56m-saaaa-aaaag-aacda-cai the wallet canister is in the subnet lspz2-jx4pu-k3e7p-znm7j-q4yum-ork6e-6w4q6-pijwq-znehu-4jabe-kqe

1 Like

Another interesting thing to note is it seems that all the requests that I sent before and that were timed out seem to have passed. So now I miss all the cycles that I sent trying to create those canisters, and have no way to find out canister ids that were created to retrieve at least part of the cycles that were spent…

Ah, no, I was wrong. I was able to transfer the some of the cycles from the wallet to the cycle ledger. But when I try to create a canister through the wallet, it still times out. So the problem persists.

1 Like

Something strange is really going on, all update calls to my backend canister are timing out again with:

Specified ingress_expiry not within expected range: Minimum allowed expiry: 2024-09-24 16:10:36.534197054 UTC, Maximum allowed expiry: 2024-09-24 16:16:06.534197054 UTC, Provided expiry: 2024-09-24 16:10:00 UTC

One time they are working, the next they are not.

Facing the same error and cannot deploy a canister.

Caused by: Failed to create batch: The replica returned a rejection error: reject code SysTransient, reject message Ingress message 0x8850474112464d4b11978c58d7d92851beca0f6cb50618310b28d928053295c4 timed out waiting to start executing., error code None

Subnet lspz2-jx4pu-k3e7p-znm7j-q4yum-ork6e-6w4q6-pijwq-znehu-4jabe-kqe

UPD: after 20 minutes it deployed without errors
UPD2: after another 2 minutes it hangs again on Starting batch. step

That’s the same error I had several times, I tried it again as long it’s worked.

I think it’s because of an overload or temporary overload subnet. :sleepy:

This issue with timing out on deploying frontend canisters is still persisting, my canister is on the subnet e66qm-3cydn-nkf4i-ml4rb-4ro6o-srm5s-x5hwq-hnprz-3meqp-s7vks-5qe

Is there a set of subnets affected? Is there anything being done to fix this or anything we can do on deploying?

The same issue persists: the asset canister’s initial deployment is failing.

Canister ID:

Subnet:

Facing this again on subnet e66qm-3cydn-nkf4i-ml4rb-4ro6o-srm5s-x5hwq-hnprz-3meqp-s7vks-5qe with dfx 0.24.0

Can someone from dfinity explain/fix this? Maybe @sat ?

Seems like this subnet has been under more load recently. You could try and set a compute allocation on your asset canister before your deploy, and set it back to default afterwards. Probably the update calls to upload the assets to the asset canister are not being scheduled, therefore it fails.

1 Like

Increasing compute allocation to 1% requires min 25 TC balance just to be able to deploy assets canister.

This subnet https://dashboard.internetcomputer.org/subnet/opn46-zyspe-hhmyp-4zu6u-7sbrh-dok77-m7dch-im62f-vyimr-a3n2c-4ae it constantly under load. There are spikes like every 1h.

I have backend and frontend canisters deployed on that subnet, which consumed about 2TC last year. To make sure that my backend canister works properly I need to increase compute allocation that will consume at least 0.8TC every day according to this.

I would rather migrate my canisters to other subnet. Is it possible?

Also, I wonder why these two subnets under load?
Are there miners deployed? If yes, why the subnet is loaded not evenly?
Why first subnet has max ~700 update calls per second, but second subnet only ~100(and it stop accepting update call at that load)?

https://dashboard.internetcomputer.org/subnet/e66qm-3cydn-nkf4i-ml4rb-4ro6o-srm5s-x5hwq-hnprz-3meqp-s7vks-5qe

https://dashboard.internetcomputer.org/subnet/opn46-zyspe-hhmyp-4zu6u-7sbrh-dok77-m7dch-im62f-vyimr-a3n2c-4ae

Another subnet where I cannot to deploy assets canister 6pbhf-qzpdk-kuqbr-pklfa-5ehhf-jfjps-zsj6q-57nrl-kzhpd-mu7hc-vae with the same problems and same spikes. Migrating to other subnet is not a solution then, a miner can be deployed in any subnet at any time.

It turns out that canister creation cost is 0.1TC and canister can live for years on 2TC, but I have to top it up to 25TC and do extra call before and after canister deployment. Kinda frustrating

UPD: found related topic Subnets with heavy compute load: what can you do now & next steps - #40 by Manu