Hi,
I was responsible for preparation of the site of ICPunks for the launch. While I think the launch was a big succes (9600 different principals claimed tokens), there are things that could be done better.
The launch was really wild, we did not expect such traffic. The actual amount of people trying to claim ICPunks was huge, during the launching day more than 180k people tried to reach our website.
Statistics from DFINITY also helps to see what was the route cause of the whole problem. 35k+ request per second is really high, no single server can handle such amount of requests (of course it is possible to handle such amount of trafic, however it requires a setup of several servers, loadbalancers, CDN, GeoIP and other techniques). Single server, when optimized can handle up to 10k+ requests per second.
When we are at the statistics to get a better perspective, google currently handles ~100k searches per second. Of course google searches are way more complicated than simple claiming of NFT, however we are talking about blockchain technology, which ensures that once the data was saved it will no be changed in an unauthorized way.
I understand that many people were frustrated about the launch, given the number of people trying to claim it was 1 to 18 chance of actually getting one.
I have few ideas on what can be improved on the development side to make it better for users.
- Right now there is no possibility to make load balancers, readonly nodes and advanced caching on IC so big launches will cause 504 errors (most users will not be able to reach website and be furious about it). We will not make the same mistake again and no launches at given time for many users are set in near future.
- Offload as many static data as possible to static assets, the issues that occurred on our site (broken timer, no way to click claim) was due to the fact that the site had to reach first to IC to check status and then make it possible to claim. Expect that IC will not return any data at all, or randomly failed calls
- Be prepared that IC will not respond from time to time. While we implemented retries for failed requests, right now it looks like a bad design choice. Automatic retries will increase the problem, instead other notifications and some easing techniques should be used.
- When IC returns 504 for change calls it does not mean the call has failed. During the high traffic I noticed several times that changes in data returned 504, however the change was actually made.
- The read and write calls are bound to the same rules, so if IC starts returning 504 it means all calls may fail.
- Make more notifications for users about failed calls. In other blockchains once you send your transaction it is there, it may not be processed in near future but it will not be lost. There is no similar mechanism yet in IC. Would it be possible to have different canister types? Like read-only and write-only, and synchronize data between them when it is possible. I think that we could manage much higher load of users this way. This way we could ensure that all people could put their request in queue and get information about success of failure later.
The fact that IC did not mangle the contract data, and actually enabled claiming 10k tokens even during such high load was a great success from technology point of view.
I strongly believe that given time we can develop together good practices and implementation patterns that will improve user experience for high load canisters.