Service Worker Bug? Body does not pass verification

I recently deployed an asset canister with .png files of different sizes. Accessing the canister through the service worker using ic0.app leads to an error as soon as the .png is close to 2MB.
Uploading and accessing a .png created with mkfile 1800k testing.png works fine.
Uploading and accessing a .png created with mkfile 1900k testing.png leads to the following error: Body does not pass verification.

dfx version 0.8.1

@kpeacock

1 Like

Accessing the files using raw.ic0.app works seamlessly.

Is this canister online? I can use the ic-hs library to check the certification header, to see if the bug is in the asset canister or the service worker.

1 Like

It’s running on https://mbihj-yyaaa-aaaae-qaata-cai.ic0.app/.

Everything above 1855kb doesn’t work with the service worker.

Ok, let me explain how I am debugging this, maybe someone finds it interesting.

At https://mbihj-yyaaa-aaaae-qaata-cai.ic0.app/1900kb.png it indeed says “Body does not pass verification”, while https://mbihj-yyaaa-aaaae-qaata-cai.ic0.app/1800kb.png goes through. Good, let’s focus on 1900kb.

Next I fetch that resource, using the raw URL, and get its SHA256:

~ $ curl -s https://mbihj-yyaaa-aaaae-qaata-cai.raw.ic0.app/1900kb.png|wc -c
1945600
~ $ curl -s https://mbihj-yyaaa-aaaae-qaata-cai.raw.ic0.app/1900kb.png|sha256sum -
f883650c320109b0d9b5df9df42e4727a96dc90597434bc658dbe06a9f3c1bb9  -

Now, I get the IC-Certification header, which is the thing that the service worker validates:

~ $ curl -i -s https://mbihj-yyaaa-aaaae-qaata-cai.raw.ic0.app/1900kb.png|grep -a -i ic-cert
ic-certificate: certificate=:2dn3o2R0cmVlgwGDAYMBgwJIY2FuaXN0ZXKDAYIEWCBRiDkyoHLHr3dYpIgI+A3DUFpO5RU/I+6QEHLXkK5Bp4MCSgAAAAAAkAAmAQGDAYMBgwJOY2VydGlmaWVkX2RhdGGCA1ggSFRk4NRgxKXOApjw6iw4N50RNmjjL+S2W/FHorBLbpuCBFggFeWoiLh+MVXGuZ8SgfsSoos+B6iqTvCq9I1JCtB4qkqCBFggG8DCehHFl7iJV0STE4WlltAYFirdelMf2tAm2USlcRaCBFggjrvAhUdHgCEbtKptqdLb0vkihgAa7Ye2RiTWCWTLOmKCBFggqc+IUSClMNqIxn7vxaI32zZD6qklupej8Q4J/pKulkuDAYIEWCBYDMw/GfqTzXHhpiG5f+IESqJ/Mq9RhenXrxrb+nwLIIMCRHRpbWWCA0mi5Pm61Pq11RZpc2lnbmF0dXJlWDCkJcLCxi58BY3uDOq94+jvg+lfFYiSZresgHikF3qVyRkDMLQBAau4G/eeoTS0dO1qZGVsZWdhdGlvbqJpc3VibmV0X2lkWB0tAwyvOX5EbAPOzk/QcqzOQyx1tqcAWS5z9dEzAmtjZXJ0aWZpY2F0ZVkCMdnZ96JkdHJlZYMBggRYIKmFICId5vKPFqWF26PICQmA0LBSdRyPLa/l8skpyJMPgwGDAkZzdWJuZXSDAYMBgwGDAYIEWCDt9vtZBlbQw6KGKCvaeiIpxquyPpVqHsStKdX3VHo6i4MBggRYIO1St5QNU/H6qp0kKEHpsNPk0zwpnItKQGIBUsxtWKBjgwJYHS0DDK85fkRsA87OT9ByrM5DLHW2pwBZLnP10TMCgwGDAk9jYW5pc3Rlcl9yYW5nZXOCA1gb2dn3gYJKAAAAAACQAAABAUoAAAAAAJ///wEBgwJKcHVibGljX2tleYIDWIUwgYIwHQYNKwYBBAGC3HwFAwECAQYMKwYBBAGC3HwFAwIBA2EAgFIwwtqWhyGP36Sb3asJNXjC6KELJ9af3aCX8oarzNriVBxxHjyncSAr/Js2a/YbF+nxFz18mm6qqQfM26O8hnZLwdVSQ3xJSAOOEBipxJ5EQYnNVTWON6b4ediMCiaTggRYIFWlcTk4pbY3iCP/IVG078E2HeKcYyd9A35ondYGqdvAggRYIKPbgo/RgIjIauzqHNbjxSLRyFHtJJWH5jS6OC5+WwVxggRYID0lCafWe+3yU4V2jaMmQ0je4LyUIY6UbBeGF0uoE45ngwJEdGltZYIDSby+qbPhteXUFmlzaWduYXR1cmVYMK8ZZOwiX/+PKoyN83Q+it+hc4bo/Um4FYtmQORTnsCWV99ELEZgqJ/cAz7JFAODFg==:, tree=:2dn3gwJLaHR0cF9hc3NldHODAYIEWCBpPd7hwyvGPRD+L+gQPssOg9oh5oXIvfhivneQtIX2qoMBggRYIKzZbIVGcXjtCTDU6L5twtnYx1ApB6rxeuJMR8yhFJ4SgwGCBFgg+r/H88YaYxgLygTHVhS1POtkfWWqQADrBYSuMHWdP3qDAYIEWCBYzw+ki8tOCR/BB3t6/BssPmPILgqxkzhd8nMZvPmBb4MBggRYIFM23wo1Y/VKvCk9NRSFvQduWNF3HzECLZKMZQoJbePjgwJLLzE5MDBrYi5wbmeCA1gg+INlDDIBCbDZtd+d9C5HJ6ltyQWXQ0vGWNvgap88G7k=:

There are two fields here, the tree and the certificate. The tree comes from the canister. I’ll copy the tree and convert it from Base64 to Hex using Base64 to Hex | Base64 Decode | Base64 Converter | Base64. This gives me

d9d9f783024b687474705f617373657473830182045820693ddee1c32bc63d10fe2fe8103ecb0e83da21e685c8bdf862be7790b485f6aa830182045820acd96c85467178ed0930d4e8be6dc2d9d8c7502907aaf17ae24c47cca1149e12830182045820fabfc7f3c61a63180bca04c75614b53ceb647d65aa4000eb0584ae30759d3f7a83018204582058cf0fa48bcb4e091fc1077b7afc1b2c3e63c82e0ab193385df27319bcf9816f8301820458205336df0a3563f54abc293d351485bd076e58d1771f31022d928c650a096de3e383024b2f313930306b622e706e6782035820f883650c320109b0d9b5df9df42e4727a96dc90597434bc658dbe06a9f3c1bb9

This is a CBOR value, and we can decode it with cbor.me, yielding an annotated value of

D9 D9F7                                 # tag(55799)
   83                                   # array(3)
      02                                # unsigned(2)
      4B                                # bytes(11)
         687474705F617373657473         # "http_assets"
      83                                # array(3)
         01                             # unsigned(1)
         82                             # array(2)
            04                          # unsigned(4)
            58 20                       # bytes(32)
               693DDEE1C32BC63D10FE2FE8103ECB0E83DA21E685C8BDF862BE7790B485F6AA # "i=\xDE\xE1\xC3+\xC6=\x10\xFE/\xE8\x10>\xCB\x0E\x83\xDA!\xE6\x85\xC8\xBD\xF8b\xBEw\x90\xB4\x85\xF6\xAA"
         83                             # array(3)
            01                          # unsigned(1)
            82                          # array(2)
               04                       # unsigned(4)
               58 20                    # bytes(32)
                  ACD96C85467178ED0930D4E8BE6DC2D9D8C7502907AAF17AE24C47CCA1149E12 # "\xAC\xD9l\x85Fqx\xED\t0\xD4\xE8\xBEm\xC2\xD9\xD8\xC7P)\a\xAA\xF1z\xE2LG\xCC\xA1\x14\x9E\x12"
            83                          # array(3)
               01                       # unsigned(1)
               82                       # array(2)
                  04                    # unsigned(4)
                  58 20                 # bytes(32)
                     FABFC7F3C61A63180BCA04C75614B53CEB647D65AA4000EB0584AE30759D3F7A # "\xFA\xBF\xC7\xF3\xC6\x1Ac\x18\v\xCA\x04\xC7V\x14\xB5<\xEBd}e\[email protected]\x00\xEB\x05\x84\xAE0u\x9D?z"
               83                       # array(3)
                  01                    # unsigned(1)
                  82                    # array(2)
                     04                 # unsigned(4)
                     58 20              # bytes(32)
                        58CF0FA48BCB4E091FC1077B7AFC1B2C3E63C82E0AB193385DF27319BCF9816F # "X\xCF\x0F\xA4\x8B\xCBN\t\x1F\xC1\a{z\xFC\e,>c\xC8.\n\xB1\x938]\xF2s\x19\xBC\xF9\x81o"
                  83                    # array(3)
                     01                 # unsigned(1)
                     82                 # array(2)
                        04              # unsigned(4)
                        58 20           # bytes(32)
                           5336DF0A3563F54ABC293D351485BD076E58D1771F31022D928C650A096DE3E3 # "S6\xDF\n5c\xF5J\xBC)=5\x14\x85\xBD\anX\xD1w\x1F1\x02-\x92\x8Ce\n\tm\xE3\xE3"
                     83                 # array(3)
                        02              # unsigned(2)
                        4B              # bytes(11)
                           2F313930306B622E706E67 # "/1900kb.png"
                        82              # array(2)
                           03           # unsigned(3)
                           58 20        # bytes(32)
                              F883650C320109B0D9B5DF9DF42E4727A96DC90597434BC658DBE06A9F3C1BB9 # "\xF8\x83e\f2\x01\t\xB0\xD9\xB5\xDF\x9D\xF4.G'\xA9m\xC9\x05\x97CK\xC6X\xDB\xE0j\x9F<\e\xB9"

The details of this CBOR values are documented in the Interface Spec; the relevant pieces is what follows the "/1900kb.png" label; there we have a hash of F883650C320109B0D9B5DF9DF42E4727A96DC90597434BC658DBE06A9F3C1BB9. This matches the SHA256 above!

So the SHA256 matches. Since other resources do validate, it is unlikely that something else related to certification is broken (e.g. wrong certificate). So I predict the bug is in the service worker, and not in the canister.

My assumption is that the service worker doesn’t implement the streaming protocol for responses larger than 2MB correctly, but looking at the code I notice that the code isn’t buggy … it’s simply not there:

Well, the service worker was quite a last minute project, and then I heard there was significant churn in that team, so maybe that is excusable. Although it would be nice if at least we’d get a better error message here…

4 Likes

That’s a fantastic explanation! I love that you included the entire debugging process instead of just linking to the TODO on GitHub. The forum should have an ICP tipping bot to encourage more of these comments

2 Likes

Feel free to send some towards d7e9c546f38fdc3171057b08282793160ea70cfd53fe77f7ee4ffda71f66eddb.

have donated a tip to the address

1 Like

Neat, more cycles for my various demo canisters (https://ce7vw-haaaa-aaaai-aanva-cai.ic0.app/, https://edrd5-rqaaa-aaaab-qaafq-cai.ic.nomeata.de/, https://6b4pv-sqaaa-aaaah-qaava-cai.raw.ic0.app/, https://fj6bh-taaaa-aaaab-qaacq-cai.raw.ic0.app/) :slight_smile:

1 Like

Thank you for the detailed explanation, this has been very insightful! One thing I am wondering though, if I’m not mistaken the nns.ic0.app is serving certified assets bigger than 2MB, do they use a different service worker that has streaming support?

If so it would be great if this made it to the SDK @kpeacock

The NNS App isn’t using dfx’s dynamic asset canister, but rather has the assets hard-coded in it’s single backend canister, with it’s own implementation of the http_request interface. And as you can see there, it doesn’t even know about streaming:

So how does it handle assets larger than 2MB (if it even does)?

Turns out that the maximum response size has been increased multiple times since the asset canister has been created (mostly in response to the NNS canisters themselves growing larger than 2MB, I believe, given that there is no way to send larger request to canister_install). But nobody told the asset canister about the increased limit, and there is no dynamic discovery of that limit, so the asset canister still tries to stream them, which then confused the service worker.

1 Like