I recently deployed an asset canister with .png files of different sizes. Accessing the canister through the service worker using ic0.app leads to an error as soon as the .png is close to 2MB.
Uploading and accessing a .png created with mkfile 1800k testing.png works fine.
Uploading and accessing a .png created with mkfile 1900k testing.png leads to the following error: Body does not pass verification.
The details of this CBOR values are documented in the Interface Spec; the relevant pieces is what follows the "/1900kb.png" label; there we have a hash of F883650C320109B0D9B5DF9DF42E4727A96DC90597434BC658DBE06A9F3C1BB9. This matches the SHA256 above!
So the SHA256 matches. Since other resources do validate, it is unlikely that something else related to certification is broken (e.g. wrong certificate). So I predict the bug is in the service worker, and not in the canister.
My assumption is that the service worker doesn’t implement the streaming protocol for responses larger than 2MB correctly, but looking at the code I notice that the code isn’t buggy … it’s simply not there:
Well, the service worker was quite a last minute project, and then I heard there was significant churn in that team, so maybe that is excusable. Although it would be nice if at least we’d get a better error message here…
That’s a fantastic explanation! I love that you included the entire debugging process instead of just linking to the TODO on GitHub. The forum should have an ICP tipping bot to encourage more of these comments
Thank you for the detailed explanation, this has been very insightful! One thing I am wondering though, if I’m not mistaken the nns.ic0.app is serving certified assets bigger than 2MB, do they use a different service worker that has streaming support?
If so it would be great if this made it to the SDK @kpeacock
The NNS App isn’t using dfx’s dynamic asset canister, but rather has the assets hard-coded in it’s single backend canister, with it’s own implementation of the http_request interface. And as you can see there, it doesn’t even know about streaming:
So how does it handle assets larger than 2MB (if it even does)?
Turns out that the maximum response size has been increased multiple times since the asset canister has been created (mostly in response to the NNS canisters themselves growing larger than 2MB, I believe, given that there is no way to send larger request to canister_install). But nobody told the asset canister about the increased limit, and there is no dynamic discovery of that limit, so the asset canister still tries to stream them, which then confused the service worker.
Yeah, we’re figuring out who will permanently own the serviceworker and certified asset canister right now, sorry for the lack of responsiveness. I also encountered the limits while I was working on my IC Canvas side project
Service worker dev here: This is very close to being resolved (finally). With the switch over to boundary nodes as VMs we will roll out a new service worker that does do streaming. Expect this to be available on the IC at some point in June.