Minting had to be paused about 80% of the way through, one of my transactions failed earlier than that, and it seems like burst traffic got the better of Entrepot’s subnet.
Just wanted to make the team aware of this – Toniq Labs is doing the mint, so best to talk to them.
There are going to be a zillion of these types of events, so hopefully the team can iterate some improvements on the bottleneck issue with burst traffic sooner rather than later.
As John and I replied in the other thread, besides query call rate limiting there was something else going on here.
We observed some update calls were quite expensive (they consumed a lot of cycles). So if they were targeting a single canister, it will take some time for each message to be processed sequential (which is the nature of update calls). We did observe a backlog was built up and some update calls eventually expired before they had a chance to be executed.
That said, out of 10k ~ 20k update calls that were received by this subnet during the peak time, only 60 timed out and expired. The subnet also stayed healthy in the meanwhile, there wasn’t too much of an impact in terms of the overall handling of update calls, despite one canister might have had some back log.
Thanks for the report, and we will continue investigate and post our findings here.