I was uploading mp3 files to an asset canister for the latest episode of my podcast Demergence, and I ran into an error I had never run into before when uploading assets:
The previous mp3 files I have uploaded were 34.1MB and 33.8MB. This file was 66.7MB. I am not sure why I would run into this error when the sdk is chunking the upload already. I would love to see this fixed. For now I can downgrade the audio file size.
The assets canister recently went though a complete rewrite to support certification feature: it adds IC-Certificate header that allows the requester to check the authenticity of the contents. All the requests to <subdomain>.ic0.app go through a service worker that does the validation. Having this build into the default assets canister allows devs to benefit from this feature without having to understand how certification works.
When all the chunks of an asset are uploaded, the new canister recomputes the SHA-256 checksum of the whole asset to validate it against the one specified by the caller before inserting the hash into the certification tree.
It’s hard to know for sure (canister debugging and profiling features are still very rudimentary), but my guess is that recomputing that hash for a 66.7MiB file is too expensive to do in one go.
I see a few ways to solve this problem:
Unconditionally trust the SHA-256 hash provided by the caller and trap if it’s not set. This might lead to hard to debug issues, but is probably safe as long as all the uploading is done by DFX.
Make the code a bit smarter and spread the hash computation over multiple rounds of execution.
I’ll create an issue to handle this case.
Note that certification requires the whole asset to be downloaded by the service worker before it’s validated, so if you go through the validated <subdomain>.ic0.app path and not <subdomain>.raw.ic0.app, using large asset files might result in poor user experience.
Thanks for the detailed response. I’m currently using <subdomain>.raw.ic0.app for everything, as unfortunately the whole service worker validation thing seems to have been a bad user experience for me thus far.
I suspect this is why the latest episode cut off just as you started talking about security. Not sure if I got a cached copy before you tried to upload a smaller version.
Interesting, that is strange to me because I wouldn’t think the asset canister had uploaded everything if it failed…perhaps it is not atomic? I thought it would either upload all of the files or fail
I just ran into this error. I was deploying hundreds of separate files. Maybe 50 MB total deploy size. It would successfully upload all static assets, but then in the final cryptographic hash computation it would fail (or whatever happens after uploading all static assets). And it would use a lot of cycles with the deploy (maybe 7.5T cycles per deploy attempt).
Is this a “too many files” and “too big of a deploy” kind of problem? To solve it, I had to deploy multiple times, iteratively adding files as I went.
When I uploaded many small svg and small js files, I also noticed my cycles being burned. Did not hit the limit but, definitely saw the cycles going down. Therefore I would guess it is a “too many files” issue.
I also got the error “exceeded the cycles limit for single message execution”
I posted multiple [Nat8] arrays with around 1 MB of data. The limit came around 200 MB and is easily reproducible.
Is there a configuration for this cycle limit?
I think this limit makes much sense in security critical applications where you would like to block brute forcing attacks and so on. However, when uploading images, the limit is achieved enormously fast.
I tried it but it results in the same error. I’m wondering how the cancan project did it. As they do the upload stream also with [Nat8] array chunks. However it would be great to know the formula behind the function cycle restrictions. Maybe there is a way to omit it. E.g. by sending smaller chunks?
This restriction may be lifted once deterministic time slicing is implemented. I would love a DFINITY engineer to give us more insight into what deterministic time slicing will do for the cycle limit
What is your canister doing with the arrays when it receives them?
My guess is that your canister is storing them in a growing data structure and after you store 200 of such arrays the garbage collection becomes so expensive that it exceeds the cycle limit. That’s why you have to store them as Blob and not as [Nat8]. For the garbage collector a 1 MB Blob is one object whereas a 1 MB [Nat8] is 1 million objects.
The easiest is to right away receive the arrays as Blob.
If easy to arrange (eg. Blobs are never deleted and don’t need fancy memory management) you could also store the blobs directly in stable memory using the ExperimentalStableMemory library and get them out of the way of the GC altogether.
This would also make upgrades less likely to run out of cycles.