Canister frontend fails to re-validate and load files from browser cache

Hi!

We currently have files used in our Unity app that are cached in the browser to allow for faster load times after the first load.

The files get cached in the browser correctly but the canister doesn’t seem to re-validate the files correctly.

This is the message that confirms that the file is cached in the browser:
[UnityCache] 'https://<CanisterID>.raw.ic0.app/ProdBuild/Build/ProdBuild.data.br' successfully downloaded and stored in the indexedDB cache

But when you reload the page, it doesn’t seem to re-validate the file, so it displays the same message because it has to re-download the file again.

When hosting on AWS, after the first load, this is the message that displays to let you know your file was re-validated and loaded from browser cache:

[UnityCache] 'https://gameurl.com/ProdBuild/Build/ProdBuild.data.br' successfully revalidated and served from the indexedDB cache

Is there a way to have the file be re-validated correctly when hosting in a canister?

Hello!

Is it possible for you to share some URLs of this Unity code hosted on the IC and on AWS?

I’d like to try and debug this, compare how the two different hosting solutions are behaving and try to learn more about how this caching mechanism works, then I may be able to pin point what is preventing the cache from working on the IC.

Hey Nathanos,

I DMed you the canister and AWS urls. If anyone else wants them feel free to reach out and I’ll sent them privately :slight_smile:

Thanks for sharing. So when I try the IC canister version, the files don’t load and throw this error:

http_request.ts:307 Failed to fetch response: Error: Unsupported encoding: "br"
    at http_request.ts:130:19
    at http_request.ts:282:30

Are you also experiencing that error?

I’m not experiencing that error. Are you using a browser that supports brotli compression? I use Chrome

Oh I know why you’re getting that error. You have to load the canister url using the “raw” keyword in the url. I’m not sure why, but it’s been the case for all Unity projects on the IC. I think it’s because the files are too large to be certified but I’m not sure. Now I’m wondering if it’s because the default dfx asset canister doesn’t support brotli compressed files (.br) for Certified Assets

You’re right, I was using the non-raw URL and it was failing because the service worker doesn’t have support for decoding brotli compressed assets.

The “raw” URL loads indefinitely in Chrome.
In Firefox it loads and then runs into issues connecting to the server, but it’s enough for me to debug the code a bit.

So there’s this custom file loading logic in a file called ProdBuild.loader.js, I assume this comes from Unity. It makes a request for the file to see if it has changed, if the file has not changed then the server should respond with a 304: 304 Not Modified - HTTP | MDN
If it responds with a 304, then ProdBuild.loader.js will load the file from the cache, otherwise it expects the server to respond with a 200 and the file that was request.

The problem is that the IC boundary nodes always respond with a 200 and never with a 304, even if the file has not been modified. Unfortunately, I don’t think there’s any way to change this behavior right now, but there are some projects in progress that are aimed at improving support for caching headers and this may solve the problem for you.

I’ll touch base with the boundary node team to see if they have anything else to add.

Thank you for looking into it and for your detailed response. That’s unfortunate that it’s an issue with boundary nodes, and can’t be configured. In the future, this would be very crucial for us to be able to run a game on the IC without long load times. Hoping for a solution in the near future for us to be able to configure this functionality :slight_smile: :crossed_fingers: Let me know what the boundary node team says

So there was actually some work done previously to support ETag and the If-None-Match header in the asset canister, but then problems arise in how the response is verified. Currently the response is verified using the response body, so if the response body is not there then verification will fail. There is work under way to develop a new version of verification that will also verify headers, this would enable verification of responses that do not include a response body.

For reference: