Front end Assets randomly fail to load with 500, other times it loads fine

Developers
1 
Failed to load resource: the server responded with a status of 500 ()

This is the exact error message.
I am confused here, since the same asset is present and nothing has changed, and after some time randomly, it does load again.

I am unable to explain this behaviour.

Any help here will be highly useful. Thanks in advance

2 Likes
2

my only guess might be that it is somehow related to cycles available? But thats a random guess

1 Like
3

I think there might be something going on with the certified data feature. For a couple of days now, on my canisters, I make a public-method query that lets me see the current certified-data:

#[query]
fn see_certified_data() -> Vec<u8> {
    data_certificate().unwrap_or_else(|| trap("can't get data certificate"))
}

This function started giving me back the strangest bytes, more than 1kb (1184-bytes yesterday, today 1404-bytes, starting with the bytes: [217, 217, 247, 163, 100, 116, … [anyone know what these bytes are?]), when the certified data shouldnt be more than 32-bytes, and it gives me back this 1kb before i even set any certified data in the canister. Also this around 1kb keeps changing when i query it, without the canister changing the certified-data. I have been trying to debug it myself to make sure i am not missing something huge but this is what i am getting. I also forked the asset canister and added the function above and reinstalled on a different canister and without uploading any assets i called this above function and it gives me back some weird ~1kb of bytes. Has anyone seen this before? Does anyone want to try this to make sure i am not the only one seeing it? put the above function into a canister and query it to see what the certified-data is.

3 Likes
4

(not letting me edit the first comment)
Looks like the canisters on the subnet yinp6 are giving back certified-data of the 1184 bytes, and the canisters on the subnet mpubz are giving back certified-data of the 1404 bytes. Both canisters are running the same wasm-module with the above public-method. The wasm-module is the certified_assets-canister with the above public-method put in.

5

Let me ping team to see if anybody has any info that can help

1 Like
6

Hi @levi,

Note that data_certificate is not the same thing as certified_data.
certified_data is a small (up to 32 bytes) blob that the canister can set in an update method.
data_certificate is a CBOR-encoded structure that contains a hash tree and a threshold signature that act as a proof that the canister indeed set its certified_data to a specific value at some point in time.

Currently, there is no System API to get just the raw certified_data back within the canister (because the canister is supposed to know its certified data anyways, and there wasn’t any compelling use case for providing this API so far).

This doesn’t explain the original behavior though. Unfortunately, it’s hard to tell where the 500 status comes from just by looking at the message. Does the request go through the service worker that validates certifications?

3 Likes
7

Hey @roman-kashitsyn. My apologies for the assumption. Turns out, when using the certified_assets-canister, if a file is set with a content_encoding: “” (empty text), the service-worker says Body does not pass verification and gives a 500. With the file encoded as a gzip and with the content-encoding set to gzip, it certifies with the success. Thank you.

Edit: I fixed it in a pull-request: This commit lets a user upload a file with a content_encoding: "",(no encoding). by levifeldman ¡ Pull Request #12 ¡ dfinity/certified-assets ¡ GitHub

2 Likes
8

When a canister sets certified_data, does it request a canister signature? Is this how the internet identity canister signs the delegation to the session key? tagging @bjoern aswell

9

Is there anything I can do from my side to avoid this issue?

I am facing the issue too where I get the service-worker says Body does not pass verification and gives a 500.

10

When I posted this question the app was broken, now its working again… no changes from our end either!

And this has been happening ever since we launched the app

11

A query call goes to just one replica, right? And isn’t the system meant to be reliable even if up to 1/3 of replicas are failing/evil? Maybe the service worker should be retrying if it gets a 500 error from a query call.

That doesn’t answer the question of why the replica is failing. I just think that it should not matter if there is a non-replicated error. What do you think?

12

What exactly do you mean by service worker?

yes, maybe adding retries should help!
thanks for the suggestions…

13

Does it go through if you refresh it a couple of times after you see a 500? Maybe try using a canister on a different subnet and see if the same thing happens.

14

It does not go through! I have seen it take a day or so for it to get back to normal!

And yes, will try using a canister from a different subnet… is there any way to specify the subnet while creating the canister ?

15

Hello @designcoderepeat

To better assist you in resolving the issue, we would need more info exact steps to reproduce the issue. Looking at the thread there are 2/3 issues being reported

  1. Asset canister
  2. Certified variables
  3. Verification failure

Please can you
Add more details, which canister, existing URL, code snippet etc

Thanks
Faraz

1 Like
16

I haven’t certified any variables…
the app works most of the time (whenever it is able to load the assets)
and other times it does not with the 500 error

for example: the app is failing now with the same error ‘body does not pass certification’
but it was working fine last night

https://rbsr6-fyaaa-aaaai-aarwa-cai.ic0.app/ is the link to the app

17

one of the assets which is not loading now, but was loading last night:

https://rbsr6-fyaaa-aaaai-aarwa-cai.ic0.app/resources/hero/guru.fbx

18

thanks - this url is failing consistently(100%) at my end.
This is a 40MB file, do you have an example for an asset that loads successfully,

19

is it possible for you to get the MD5SUM of the original file guru.fbx that you uploaded into the canister?

on linux you can do it by

md5sum /tmp/guru.fbx

can you please take a look at this post

1 Like
20

@designcoderepeat

Hey I ran the same analysis as @neometa did in the following report
‘Service Worker Bug? Body does not pass verification’

Due to lack of streaming support in the service worker, an asset greater than 1800K bytes in size will run into the body cannot be verified error.

sha256sum /tmp/guru.fbx
fddee745a1d9676221cdafe01d3168c58f6954e8c4361c178915b55dc5914f4a /tmp/guru.fbx

“/resources/hero/guru.fbx”

                                         82 # array(2)
                                            03 # unsigned(3)
                                            58 20 # bytes(32)
                                               FDDEE745A1D9676221CDAFE01D3168C58F6954E8C4361C178915B55DC5914F4A # "\xFD\xDE\xE7E\xA1\xD9gb!

CBOR dump of guru.fbx

1 Like