I recently deployed an asset canister with .png files of different sizes. Accessing the canister through the service worker using ic0.app leads to an error as soon as the .png is close to 2MB.
Uploading and accessing a .png created with mkfile 1800k testing.png works fine.
Uploading and accessing a .png created with mkfile 1900k testing.png leads to the following error: Body does not pass verification.
dfx version 0.8.1
Accessing the files using raw.ic0.app works seamlessly.
Is this canister online? I can use the ic-hs library to check the certification header, to see if the bug is in the asset canister or the service worker.
It’s running on https://mbihj-yyaaa-aaaae-qaata-cai.ic0.app/.
Everything above 1855kb doesn’t work with the service worker.
Ok, let me explain how I am debugging this, maybe someone finds it interesting.
At https://mbihj-yyaaa-aaaae-qaata-cai.ic0.app/1900kb.png it indeed says “Body does not pass verification”, while https://mbihj-yyaaa-aaaae-qaata-cai.ic0.app/1800kb.png goes through. Good, let’s focus on 1900kb.
Next I fetch that resource, using the raw URL, and get its SHA256:
~ $ curl -s https://mbihj-yyaaa-aaaae-qaata-cai.raw.ic0.app/1900kb.png|wc -c
1945600
~ $ curl -s https://mbihj-yyaaa-aaaae-qaata-cai.raw.ic0.app/1900kb.png|sha256sum -
f883650c320109b0d9b5df9df42e4727a96dc90597434bc658dbe06a9f3c1bb9 -
Now, I get the IC-Certification header, which is the thing that the service worker validates:
~ $ curl -i -s https://mbihj-yyaaa-aaaae-qaata-cai.raw.ic0.app/1900kb.png|grep -a -i ic-cert
ic-certificate: certificate=:2dn3o2R0cmVlgwGDAYMBgwJIY2FuaXN0ZXKDAYIEWCBRiDkyoHLHr3dYpIgI+A3DUFpO5RU/I+6QEHLXkK5Bp4MCSgAAAAAAkAAmAQGDAYMBgwJOY2VydGlmaWVkX2RhdGGCA1ggSFRk4NRgxKXOApjw6iw4N50RNmjjL+S2W/FHorBLbpuCBFggFeWoiLh+MVXGuZ8SgfsSoos+B6iqTvCq9I1JCtB4qkqCBFggG8DCehHFl7iJV0STE4WlltAYFirdelMf2tAm2USlcRaCBFggjrvAhUdHgCEbtKptqdLb0vkihgAa7Ye2RiTWCWTLOmKCBFggqc+IUSClMNqIxn7vxaI32zZD6qklupej8Q4J/pKulkuDAYIEWCBYDMw/GfqTzXHhpiG5f+IESqJ/Mq9RhenXrxrb+nwLIIMCRHRpbWWCA0mi5Pm61Pq11RZpc2lnbmF0dXJlWDCkJcLCxi58BY3uDOq94+jvg+lfFYiSZresgHikF3qVyRkDMLQBAau4G/eeoTS0dO1qZGVsZWdhdGlvbqJpc3VibmV0X2lkWB0tAwyvOX5EbAPOzk/QcqzOQyx1tqcAWS5z9dEzAmtjZXJ0aWZpY2F0ZVkCMdnZ96JkdHJlZYMBggRYIKmFICId5vKPFqWF26PICQmA0LBSdRyPLa/l8skpyJMPgwGDAkZzdWJuZXSDAYMBgwGDAYIEWCDt9vtZBlbQw6KGKCvaeiIpxquyPpVqHsStKdX3VHo6i4MBggRYIO1St5QNU/H6qp0kKEHpsNPk0zwpnItKQGIBUsxtWKBjgwJYHS0DDK85fkRsA87OT9ByrM5DLHW2pwBZLnP10TMCgwGDAk9jYW5pc3Rlcl9yYW5nZXOCA1gb2dn3gYJKAAAAAACQAAABAUoAAAAAAJ///wEBgwJKcHVibGljX2tleYIDWIUwgYIwHQYNKwYBBAGC3HwFAwECAQYMKwYBBAGC3HwFAwIBA2EAgFIwwtqWhyGP36Sb3asJNXjC6KELJ9af3aCX8oarzNriVBxxHjyncSAr/Js2a/YbF+nxFz18mm6qqQfM26O8hnZLwdVSQ3xJSAOOEBipxJ5EQYnNVTWON6b4ediMCiaTggRYIFWlcTk4pbY3iCP/IVG078E2HeKcYyd9A35ondYGqdvAggRYIKPbgo/RgIjIauzqHNbjxSLRyFHtJJWH5jS6OC5+WwVxggRYID0lCafWe+3yU4V2jaMmQ0je4LyUIY6UbBeGF0uoE45ngwJEdGltZYIDSby+qbPhteXUFmlzaWduYXR1cmVYMK8ZZOwiX/+PKoyN83Q+it+hc4bo/Um4FYtmQORTnsCWV99ELEZgqJ/cAz7JFAODFg==:, tree=:2dn3gwJLaHR0cF9hc3NldHODAYIEWCBpPd7hwyvGPRD+L+gQPssOg9oh5oXIvfhivneQtIX2qoMBggRYIKzZbIVGcXjtCTDU6L5twtnYx1ApB6rxeuJMR8yhFJ4SgwGCBFgg+r/H88YaYxgLygTHVhS1POtkfWWqQADrBYSuMHWdP3qDAYIEWCBYzw+ki8tOCR/BB3t6/BssPmPILgqxkzhd8nMZvPmBb4MBggRYIFM23wo1Y/VKvCk9NRSFvQduWNF3HzECLZKMZQoJbePjgwJLLzE5MDBrYi5wbmeCA1gg+INlDDIBCbDZtd+d9C5HJ6ltyQWXQ0vGWNvgap88G7k=:
There are two fields here, the tree and the certificate. The tree comes from the canister. I’ll copy the tree and convert it from Base64 to Hex using Base64 to Hex | Base64 Decode | Base64 Converter | Base64. This gives me
d9d9f783024b687474705f617373657473830182045820693ddee1c32bc63d10fe2fe8103ecb0e83da21e685c8bdf862be7790b485f6aa830182045820acd96c85467178ed0930d4e8be6dc2d9d8c7502907aaf17ae24c47cca1149e12830182045820fabfc7f3c61a63180bca04c75614b53ceb647d65aa4000eb0584ae30759d3f7a83018204582058cf0fa48bcb4e091fc1077b7afc1b2c3e63c82e0ab193385df27319bcf9816f8301820458205336df0a3563f54abc293d351485bd076e58d1771f31022d928c650a096de3e383024b2f313930306b622e706e6782035820f883650c320109b0d9b5df9df42e4727a96dc90597434bc658dbe06a9f3c1bb9
This is a CBOR value, and we can decode it with cbor.me, yielding an annotated value of
D9 D9F7 # tag(55799)
83 # array(3)
02 # unsigned(2)
4B # bytes(11)
687474705F617373657473 # "http_assets"
83 # array(3)
01 # unsigned(1)
82 # array(2)
04 # unsigned(4)
58 20 # bytes(32)
693DDEE1C32BC63D10FE2FE8103ECB0E83DA21E685C8BDF862BE7790B485F6AA # "i=\xDE\xE1\xC3+\xC6=\x10\xFE/\xE8\x10>\xCB\x0E\x83\xDA!\xE6\x85\xC8\xBD\xF8b\xBEw\x90\xB4\x85\xF6\xAA"
83 # array(3)
01 # unsigned(1)
82 # array(2)
04 # unsigned(4)
58 20 # bytes(32)
ACD96C85467178ED0930D4E8BE6DC2D9D8C7502907AAF17AE24C47CCA1149E12 # "\xAC\xD9l\x85Fqx\xED\t0\xD4\xE8\xBEm\xC2\xD9\xD8\xC7P)\a\xAA\xF1z\xE2LG\xCC\xA1\x14\x9E\x12"
83 # array(3)
01 # unsigned(1)
82 # array(2)
04 # unsigned(4)
58 20 # bytes(32)
FABFC7F3C61A63180BCA04C75614B53CEB647D65AA4000EB0584AE30759D3F7A # "\xFA\xBF\xC7\xF3\xC6\x1Ac\x18\v\xCA\x04\xC7V\x14\xB5<\xEBd}e\xAA@\x00\xEB\x05\x84\xAE0u\x9D?z"
83 # array(3)
01 # unsigned(1)
82 # array(2)
04 # unsigned(4)
58 20 # bytes(32)
58CF0FA48BCB4E091FC1077B7AFC1B2C3E63C82E0AB193385DF27319BCF9816F # "X\xCF\x0F\xA4\x8B\xCBN\t\x1F\xC1\a{z\xFC\e,>c\xC8.\n\xB1\x938]\xF2s\x19\xBC\xF9\x81o"
83 # array(3)
01 # unsigned(1)
82 # array(2)
04 # unsigned(4)
58 20 # bytes(32)
5336DF0A3563F54ABC293D351485BD076E58D1771F31022D928C650A096DE3E3 # "S6\xDF\n5c\xF5J\xBC)=5\x14\x85\xBD\anX\xD1w\x1F1\x02-\x92\x8Ce\n\tm\xE3\xE3"
83 # array(3)
02 # unsigned(2)
4B # bytes(11)
2F313930306B622E706E67 # "/1900kb.png"
82 # array(2)
03 # unsigned(3)
58 20 # bytes(32)
F883650C320109B0D9B5DF9DF42E4727A96DC90597434BC658DBE06A9F3C1BB9 # "\xF8\x83e\f2\x01\t\xB0\xD9\xB5\xDF\x9D\xF4.G'\xA9m\xC9\x05\x97CK\xC6X\xDB\xE0j\x9F<\e\xB9"
The details of this CBOR values are documented in the Interface Spec; the relevant pieces is what follows the "/1900kb.png" label; there we have a hash of F883650C320109B0D9B5DF9DF42E4727A96DC90597434BC658DBE06A9F3C1BB9. This matches the SHA256 above!
So the SHA256 matches. Since other resources do validate, it is unlikely that something else related to certification is broken (e.g. wrong certificate). So I predict the bug is in the service worker, and not in the canister.
My assumption is that the service worker doesn’t implement the streaming protocol for responses larger than 2MB correctly, but looking at the code I notice that the code isn’t buggy … it’s simply not there:
Well, the service worker was quite a last minute project, and then I heard there was significant churn in that team, so maybe that is excusable. Although it would be nice if at least we’d get a better error message here…
That’s a fantastic explanation! I love that you included the entire debugging process instead of just linking to the TODO on GitHub. The forum should have an ICP tipping bot to encourage more of these comments
Feel free to send some towards d7e9c546f38fdc3171057b08282793160ea70cfd53fe77f7ee4ffda71f66eddb.
have donated a tip to the address
Neat, more cycles for my various demo canisters (https://ce7vw-haaaa-aaaai-aanva-cai.ic0.app/, https://edrd5-rqaaa-aaaab-qaafq-cai.ic.nomeata.de/, https://6b4pv-sqaaa-aaaah-qaava-cai.raw.ic0.app/, https://fj6bh-taaaa-aaaab-qaacq-cai.raw.ic0.app/) 
Thank you for the detailed explanation, this has been very insightful! One thing I am wondering though, if I’m not mistaken the nns.ic0.app is serving certified assets bigger than 2MB, do they use a different service worker that has streaming support?
If so it would be great if this made it to the SDK @kpeacock
The NNS App isn’t using dfx’s dynamic asset canister, but rather has the assets hard-coded in it’s single backend canister, with it’s own implementation of the http_request interface. And as you can see there, it doesn’t even know about streaming:
So how does it handle assets larger than 2MB (if it even does)?
Turns out that the maximum response size has been increased multiple times since the asset canister has been created (mostly in response to the NNS canisters themselves growing larger than 2MB, I believe, given that there is no way to send larger request to canister_install). But nobody told the asset canister about the increased limit, and there is no dynamic discovery of that limit, so the asset canister still tries to stream them, which then confused the service worker.
Yeah, we’re figuring out who will permanently own the serviceworker and certified asset canister right now, sorry for the lack of responsiveness. I also encountered the limits while I was working on my IC Canvas side project
Very helpful! We save the encrypted data in ICP but But our team is most likely to give up streaming from ICP at this moment.
Service worker dev here: This is very close to being resolved (finally). With the switch over to boundary nodes as VMs we will roll out a new service worker that does do streaming. Expect this to be available on the IC at some point in June.
Hi, Frederik!
Your appearance is always associated with good news. Thank you so much! 
Ah, has streaming been implemented - I’m trying to submit a OpenCV with machine learning models Unity webgl app to Supernova and running into this issue on main net.
Unfortunately, the new boundary node VMs have not been rolled out yet. But the go live should be imminent. After that, streaming will be available.
Maybe @martin_DFN1 can give a more detailed status update.
We’re testing the new boundary node VMs at the moment and are ironing out the last bugs. We want to be certain that everything works.
I’m so hype for streaming support to roll out!
Is there any info you can give - how soon might this be?
I’ve hit the same issue loading a JSON which was over 2mb