Hi, this is the developer from DFinance, we launched our testnet today, it all went smoothly until a while ago, users talking about loading too slow, I tried to call our backend canister with dfx, but I got 504 error, is this because of too much traffic? I thought IC has load balance?
Canister id: Principal lf23w-ciaaa-aaaah-qaeya-cai | ic.rocks
We are in this subnet: Subnet pjljw-kztyl-46ud4-ofrj6-nzkhm-3n4nt-wi3jt-ypmav-ijqkt-gjf66-uae | ic.rocks
I believe this is a problem for your subnet, other projects reported the same issues, maybe youāre all on the same subnet? (entrepot & icpunks)
Yes I believe so, we are in the same subnet.
gg, you broke the IC!
Haha, stress test, find the problem early so we can grow stronger.
It seems that the current subnets architecture can not handle too much triffic.
Luckily the IC can scale and add nodes to the subnets 
!
Weāve also launched a whitelist event which involved mutiple people to login to our website icpunks.com and had exact problem. It looks like that we did a stress test 
About 9k user participated in our testnet event, at the first several hours, itās working smoothly, then it stucked.
So you are the culprit ;). Iām glad that it happend today and not on the claiming day :D.
As far as we can tell this has to do with the fact that subnet pjljw handles by far the most queries, combined with an issue we discovered recently, where we only cache the compiled Wasm when executing updates, not queries. Meaning that before an update is executed on a canister, each query will compile the Wasm from scratch. Because of the load, some queries will be really slow, while other queries will time out before they even get a chance to execute.
The reason why this started suddenly is that we upgraded the replica version on subnet pjljw a few hours ago and and caches got purged. Why this behavior wasnāt noticed until now (on previous replica upgrades) is unclear.
Weāre working on a fix, but it may take a while to deploy. In the meantime, the more canisters on pjljw handle at least one update, the less contention among queries, so this should become better over time.
Thanks for the fast response. Is there anything that we can do right now, or just wait?
Make sure to run an update query on each of your canisters. (o:
More seriously though, I donāt think thereās anything you can do. Iāll try to figure out if I can find a way to run a replicated query (i.e. run a query via an ingress message) on all canisters on the subnet, to prime the cache. Actually, now that I think of it, anyone could do it.
We have same problem accessing our application which runs in the same subnet.
IC Drive was also affected, the app was loading super slow and our public file links were also affected. Seems itās fine now.
The issue seems to still happen, I face the net::ERR_ABORTED 504 with my asset canister too right now.
Yes, it happens again
We noticed the issue happened again. The root cause has been identified and the fix is on the way. Apologies for the inconvenience and thanks for your patience.
Hi Shuo, do you know when it will be fixed? Itās getting hard to test anything
Thanks for asking, Iāve pinged the team to see who can give an update. Sorry I canāt help, Iām not familiar with the status.
