Canister ic3o3-qiaaa-aaaae-qaaia-cai exceeded the cycles limit for single message execution

I was uploading mp3 files to an asset canister for the latest episode of my podcast Demergence, and I ran into an error I had never run into before when uploading assets:

Deploying all canisters.
All canisters have already been created.
Building canisters...
Installing canisters...
Upgrading code for canister assets-0, with canister_id ic3o3-qiaaa-aaaae-qaaia-cai
Authorizing our identity (lastmjs-mercury-0) to the asset canister...
Uploading assets to asset canister...
  /demergence.rss 1/1 (2869 bytes)
  /episodes/demergence-0.mp3 (34071925 bytes) sha 345dc9e344be6abc5d47db303a4a3ebb7291dd61b8aefdcf7898e5b6cf8e32a9 is already installed
  /episodes/demergence-1.mp3 (33805254 bytes) sha 573a8d271f3a02113c657d19e23f873de6dcc9744abeb66b029bb233b24f0120 is already installed
  /episodes/demergence-2.mp3 1/36 (1900000 bytes)
  /episodes/demergence-2.mp3 2/36 (1900000 bytes)
  /episodes/demergence-2.mp3 3/36 (1900000 bytes)
  /episodes/demergence-2.mp3 4/36 (1900000 bytes)
  /episodes/demergence-2.mp3 5/36 (1900000 bytes)
  /episodes/demergence-2.mp3 6/36 (1900000 bytes)
  /episodes/demergence-2.mp3 7/36 (1900000 bytes)
  /episodes/demergence-2.mp3 8/36 (1900000 bytes)
  /episodes/demergence-2.mp3 9/36 (1900000 bytes)
  /episodes/demergence-2.mp3 10/36 (1900000 bytes)
  /episodes/demergence-2.mp3 11/36 (1900000 bytes)
  /episodes/demergence-2.mp3 12/36 (1900000 bytes)
  /episodes/demergence-2.mp3 13/36 (1900000 bytes)
  /episodes/demergence-2.mp3 14/36 (1900000 bytes)
  /episodes/demergence-2.mp3 15/36 (1900000 bytes)
  /episodes/demergence-2.mp3 16/36 (1900000 bytes)
  /episodes/demergence-2.mp3 17/36 (1900000 bytes)
  /episodes/demergence-2.mp3 18/36 (1900000 bytes)
  /episodes/demergence-2.mp3 19/36 (1900000 bytes)
  /episodes/demergence-2.mp3 20/36 (1900000 bytes)
  /episodes/demergence-2.mp3 21/36 (1900000 bytes)
  /episodes/demergence-2.mp3 22/36 (1900000 bytes)
  /episodes/demergence-2.mp3 23/36 (1900000 bytes)
  /episodes/demergence-2.mp3 24/36 (1900000 bytes)
  /episodes/demergence-2.mp3 25/36 (1900000 bytes)
  /episodes/demergence-2.mp3 26/36 (1900000 bytes)
  /episodes/demergence-2.mp3 27/36 (1900000 bytes)
  /episodes/demergence-2.mp3 28/36 (1900000 bytes)
  /episodes/demergence-2.mp3 29/36 (1900000 bytes)
  /episodes/demergence-2.mp3 30/36 (1900000 bytes)
  /episodes/demergence-2.mp3 31/36 (1900000 bytes)
  /episodes/demergence-2.mp3 32/36 (1900000 bytes)
  /episodes/demergence-2.mp3 33/36 (1900000 bytes)
  /episodes/demergence-2.mp3 34/36 (1900000 bytes)
  /episodes/demergence-2.mp3 35/36 (1900000 bytes)
  /episodes/demergence-2.mp3 36/36 (244436 bytes)
  /index.html 1/1 (1530 bytes)
  /index.html (gzip) 1/1 (565 bytes)
  /demergence.png (28290 bytes) sha 9f33c93b78d0c93f6aa81c4e25f3b5c254e52d143c8e179de34f05946b65b68b is already installed
The Replica returned an error: code 5, message: "Canister ic3o3-qiaaa-aaaae-qaaia-cai exceeded the cycles limit for single message execution."

The previous mp3 files I have uploaded were 34.1MB and 33.8MB. This file was 66.7MB. I am not sure why I would run into this error when the sdk is chunking the upload already. I would love to see this fixed. For now I can downgrade the audio file size.

1 Like

I’ve alerted folk internally…

3 Likes

I assume you use a fresh version of dfx.

The assets canister recently went though a complete rewrite to support certification feature: it adds IC-Certificate header that allows the requester to check the authenticity of the contents. All the requests to <subdomain>.ic0.app go through a service worker that does the validation. Having this build into the default assets canister allows devs to benefit from this feature without having to understand how certification works.

When all the chunks of an asset are uploaded, the new canister recomputes the SHA-256 checksum of the whole asset to validate it against the one specified by the caller before inserting the hash into the certification tree.

It’s hard to know for sure (canister debugging and profiling features are still very rudimentary), but my guess is that recomputing that hash for a 66.7MiB file is too expensive to do in one go.

I see a few ways to solve this problem:

  1. Unconditionally trust the SHA-256 hash provided by the caller and trap if it’s not set. This might lead to hard to debug issues, but is probably safe as long as all the uploading is done by DFX.
  2. Make the code a bit smarter and spread the hash computation over multiple rounds of execution.

I’ll create an issue to handle this case.

Note that certification requires the whole asset to be downloaded by the service worker before it’s validated, so if you go through the validated <subdomain>.ic0.app path and not <subdomain>.raw.ic0.app, using large asset files might result in poor user experience.

6 Likes

Thanks for the detailed response. I’m currently using <subdomain>.raw.ic0.app for everything, as unfortunately the whole service worker validation thing seems to have been a bad user experience for me thus far.

1 Like

I suspect this is why the latest episode cut off just as you started talking about security. Not sure if I got a cached copy before you tried to upload a smaller version.

1 Like

I just managed to re-download and it went from something like 31m36s to 51m17s

2 Likes

Interesting, that is strange to me because I wouldn’t think the asset canister had uploaded everything if it failed…perhaps it is not atomic? I thought it would either upload all of the files or fail

Deployment of assets is an atomic operation: if the commit fails, the incomplete assets aren’t used for serving.

1 Like

I just ran into this error. I was deploying hundreds of separate files. Maybe 50 MB total deploy size. It would successfully upload all static assets, but then in the final cryptographic hash computation it would fail (or whatever happens after uploading all static assets). And it would use a lot of cycles with the deploy (maybe 7.5T cycles per deploy attempt).

Is this a “too many files” and “too big of a deploy” kind of problem? To solve it, I had to deploy multiple times, iteratively adding files as I went.

1 Like

I had the same problem as @bob11, It was due to one mp4 file that was 87.7 MB. Uploading a smaller file of less than 2 MB fixed the issue.

When I uploaded many small svg and small js files, I also noticed my cycles being burned. Did not hit the limit but, definitely saw the cycles going down. Therefore I would guess it is a “too many files” issue.