We had assets available on-chain for a long time, but today we faced ERR_HTTP2_PROTOCOL_ERROR issue when trying to access the files… Here is a link with an example: https://nszbk-7iaaa-aaaap-abczq-cai.raw.icp0.io/-/64ba621a76bd639aa6484af4/-/logo.png, is there any issues going on at the moment and when this is expected to be resolved?
Hey @3oltan,
we are not aware of any issue right now, but we are looking into it as we have yesterday rolled out a new release.
Here is another example for the file: https://nszbk-7iaaa-aaaap-abczq-cai.raw.icp0.io/-/64ba621a76bd639aa6484af4/-/p4.png this one is not loading properly but is accessible. We also can access the same files through prptl.io like: https://prptl.io/-/nszbk-7iaaa-aaaap-abczq-cai/-/64ba621a76bd639aa6484af4/-/p4.png
1 Like
Hey @3oltan
we looked into it and realized that the canister returns range headers that do not follow the spec:
# curl -v -r 0-1048576 --unix-socket /run/ic-node/icx-proxy.socket http://nszbk-7iaaa-aaaap-abczq-cai.raw.icp0.io/-/64ba621a76bd639aa6484af4/-/logo.png
...
< accept-ranges: bytes
< content-range: bytes 0-1048576/58128
Where the expected content-range should look like Content-Range: <unit> <range-start>-<range-end>/<size> (see here).
Are you all now sending down that you accept ranges where as before you were not?
We have two streaming strategies where one handled range requests manually and only worked via our proxy. You can find the code here: origyn_nft/src/origyn_nft_reference/http.mo at main · ORIGYN-SA/origyn_nft · GitHub
Exactly, what changed on the boundary node side is that now they are making range requests and before they weren’t.
Why would you make a range request if it wasn’t asked for by the client.
Shouldn’t the end of the range in this line here here should be according to the actual content being served instead of specifying the end requested by the client?
You can see where we split the streaming strategy here based on whether we find a range request or not. We DO NOT expect a range request under most circumstances so that we can use certification for those files. No range request = classic streaming strategy. If there is a range request we handle it completely differently. origyn_nft/src/origyn_nft_reference/http.mo at bf874a83d8ac270fd21d883acb9b675a1c8124da · ORIGYN-SA/origyn_nft · GitHub
Indeed, the slicing works well for raw, but does not support certification just like that. We will roll back.
Happy to work through getting this all to work and update what we’re doing but may need some time to test/adjust.
Can you spot where we are not adhering to the spec?
On the line you pointed out we return our rEnd because that is the just one chunk path way. We store files in chunks of 2MB. So if it is less than one chunk the math is a bit easier. These files that arent’ working are likely bigger than 2MB and thus taking the other pathway.
The fix for now: We disable it on non-raw as with certification it will not work. We keep it for raw. Sorry for the problems!
Regarding where the error is:
The problem is that whatever range is request, the canister sets then end of the range to the requested range end, whereas it should set it to min(actual file end and requested range end). It doesn’t matter if the response is larger than 2MB or not.
The boundary node requests chunks of 1MB. You can see that the request fails for https://nszbk-7iaaa-aaaap-abczq-cai.raw.icp0.io/-/64ba621a76bd639aa6484af4/-/logo.png because the file is smaller than 1MB and the header is wrong.
However, it works for the first chunk of https://nszbk-7iaaa-aaaap-abczq-cai.raw.icp0.io/-/64ba621a76bd639aa6484af4/-/p4.png as the file is larger than 1MB and the header will only be wrong for the second range request as the chunk is smaller than the requested range.
We had to do all of this hoop jumping because safari and apple do strange things with movie files. If I recall, they ask for the file normally and then they cut the connection if they see it is a media file, then request 1 byte to get the size then start sending range requests. I see that I return accept-ranges here on our Large content(classic streaming strategy), so maybe this is where things get wonky? The first request for a png would likely not have a range request but then I tell you I accept ranges? Would you go back and get the second chunk a different way?
So this is NOT correct?
("Content-Range", "bytes " # Nat.toText(rStart) # "-" # Nat.toText(rEnd) # "/" # Nat.toText(size)),
Doesn’t this say that this chunk is in bytes and is the rStart to rEnd of / totalBytes?
It is correct unless the size of the chunk is smaller than the requested byte range.
With our current code, I don’t think we will ever serve you past the first chunk with a range request unless you provide us with our custom streaming strategy token.
What I think is happening now(please correct me if I’m wrong) is:
- You request a file that is greater than 2MB.
- We return the first chunk with standard streaming strategy, but I have an accept ranges header in it so you all switch to requesting chunks via range over using the classic streaming strategy.
- Your code does the range request but doesn’t have the right streaming token because I require the first request to be a range request to go down that path.
Maybe can you link to the boundary node code and I can take a look to try to resolve?
kk…So if rEnd > rSize { rEnd = rSize -1}?
1 Like
The streaming strategy is something independent of the range request:
We have two components: nginx and icx-proxy.
nginx: the range requests have been introduced here.nginxwill just request chunks of 1MB in size until it got the last chunk.nginxwill automatically keep on requesting until it got the full file. Here is no streaming strategy involved.icx-proxy: will just turn the HTTP request into a query call and will try to turn the response into an HTTP response. However, since the canister can only serve a certain response size, it will have to chunk it if the response is larger. This is done using the streaming strategy.icx-proxywill apply the streaming strategy to assemble the response.
The streaming strategy is only active if the response from the canister is larger than the maximum response size. Since nginx always requests chunks of 1MB in size, it is always smaller than the response size and will not go through the streaming strategy. If it would request chunks of size 10MB, then icx-proxy would have to use a streaming strategy.
Does that make sense or did I explain it too complicated?