Is it possible to make raw.ic0.app calls from ic0.app and reverse?

3cL1p5e7 · October 29, 2021, 7:21pm

I want to make a request from my application to another application via an unsafe API (ic0.raw.app), but I get an error. Is this the target behavior?

It seems that the Service worker did not resolve a canister ID from another domain URL.
Steps to reproduce:

Go to https://***.ic0.app
make fetch('https://***.raw.ic0.app/index.html')
Got error URL "https://***.raw.ic0.app/" did not resolve to a canister ID.

After debugging I found, that service worker unable to resolve canisterId when page hostname does not match requesting URL. Difference is between *.ic0.app and *.raw.ic0.app.

It’s blocker for me because I want to check certificated assets from raw resource before the load a resource in IFrame. And this is the last way to solve my problem after Why can’t IFrame be used with certificated URLs?

Please, help or answer

3cL1p5e7 · October 29, 2021, 7:36pm

Also, there is no way to register custom service worker on https://***.ic0.app, because any request to main domain is accompanied by bootstrap-page loading instead of a custom worker JS-source.

Will us be able to register out custom service workers in future?

nomeata · October 29, 2021, 9:09pm

I’m surprised: When accessing *.raw.ic0.app, no service worker should be involved. The point of raw is that it hits the HTTP Gateway on the boundary node and no certificate checking happen, so no reason to have a service worker involved.

3cL1p5e7 · October 30, 2021, 12:05pm

This applies to requests from the frontend, where the service worker is already installed. For example

go to any canister (for example go to OpenChat https://7e6iv-biaaa-aaaaf-aaada-cai.ic0.app/)
open the console
we make a request to the same canister in raw domain zone fetch('https://7e6iv-biaaa-aaaaf-aaada-cai.raw.ic0.app')
got error, Network tab is empty

nomeata · October 30, 2021, 12:08pm

Good analysis. That sounds like a bug in the service worker to me – it should let raw urls through.

3cL1p5e7 · October 30, 2021, 12:10pm

Roger!
Any suggestions who can help with this issue?

nomeata · October 30, 2021, 12:11pm

It should be sufficient to extend this check, to let requests to raw urls through (just like requests to unrelated domains):

github.com

dfinity/agent-js/blob/main/apps/sw-cert/src/sw/http_request.ts#L327

    
      
              } catch (e) {
                console.error('Failed to fetch response:', e);
                return new Response(`Failed to fetch response: ${e.toString()}`, { status: 500 });
              }
            }
          
          
  // Last check. IF this is not part of the same domain, then we simply let it load as is.
            // The same domain will always load using our service worker, and not the same domain
            // would load by reference. If you want security for your users at that point you
            // should use SRI to make sure the resource matches.
            if (!url.hostname.endsWith(swDomains)) {
              // todo: Do we need to check for headers and certify the content here?
              return await fetch(request);
            }
          
          
  console.error(`URL ${JSON.stringify(url.toString())} did not resolve to a canister ID.`);
            return new Response('Could not find the canister ID.', { status: 404 });
          }

3cL1p5e7 · October 30, 2021, 12:15pm

And I still could not find the sourcecode (sorry). Thank you so much for the link!
Perhaps I’ll open a pull request

3cL1p5e7 · October 30, 2021, 1:27pm

@kpeacock Check plz
Pull request

3cL1p5e7 · November 18, 2021, 6:24pm

Pull request was merged.
@bitdivine @kpeacock Thanks you so much!

kpeacock · November 18, 2021, 6:28pm

We appreciate the initiative, and thanks for bearing with us while we’re in the middle of migrating the serviceworker to a new codebase!

lastmjs · January 26, 2022, 12:01am

I just ran into this issue here: https://ic3o3-qiaaa-aaaae-qaaia-cai.ic0.app/

I am currently sharing https://ic3o3-qiaaa-aaaae-qaaia-cai.raw.ic0.app/ instead to get around this. All of the audio elements fail without .raw in the URL address bar.

3cL1p5e7 · January 31, 2022, 12:33pm

As far as I can see the audio works and it’s great!
If you want to use audio elements without .raw, try to certify uploaded files in the same way as certified-assets or cert-var. I haven’t tried it, but it should work.
Thank you for your feedback!

lastmjs · February 2, 2022, 7:48pm

I thought I responded to this, I fixed it up unregistering the service worker, I think it was out of date. Service workers often seem to have a problem staying up to date

3cL1p5e7 · February 3, 2022, 3:18pm

this happens because the audiofile is splited into chunks. When service-worker fetching chunk and trying to validate it, worker compares file hash from field sha256 with chunk hash. And of course they don’t match. When file was not chunkified, service-worker loads file at once and hashes match.

github.com

dfinity/agent-js/blob/cf407c49287d74f0e52395154925c8025eef0d5d/apps/sw-cert/src/sw/validation.ts#L73


      
              treeSha = lookup_path(['http_assets', '/index.html'], hashTree);
            }
          
            if (!treeSha) {
              // The tree returned in the certification header is wrong. Return false.
              // We don't throw here, just invalidate the request.
              console.error(`Invalid Tree in the header. Does not contain path ${JSON.stringify(path)}`);
              return false;
            }
          
            return !!treeSha && equal(sha, treeSha);
          }
          
          function equal(buf1: ArrayBuffer, buf2: ArrayBuffer): boolean {
            if (buf1.byteLength !== buf2.byteLength) {
              return false;
            }
          
            const a1 = new Uint8Array(buf1);
            const a2 = new Uint8Array(buf2);
            for (let i = 0; i < a1.length; i++) {

I plan to post a topic about it

3cL1p5e7 · February 3, 2022, 5:02pm

Or make streaming implementation on service-worker

UPDATE

kpeacock · February 3, 2022, 7:46pm

We’re a couple sprints in, and I’ve still been focused on the invoice canister. I’d still like to demonstrate chunking via agent-js, but that won’t resolve the basic issue of the serviceworker needing streaming support

3cL1p5e7 · February 3, 2022, 10:13pm

I have a similar question, not about streaming
Is it correct to use merkletree hashing algorithm while uploading chunks into certified_assets and make support of this on service-worker?

At now, not chunkified files works correctly (fetch(...), <img src...>) in the browser, but downloading chunkified files raises 500 validation error.

To fix this, I want to try:

calculate and save merklethee of chunks to sha256 field of certified_assets (by yourself)
[Service-worker improvement] make equal(sha, treeSha)-function to validate merkletree (because treeSha is a hash of chunk and merkletree is usually used for validating chunk hashes)

Ideally, this will allow to using chunkified assets at least in simple fetch or <img src=...> cases.

3cL1p5e7 · February 9, 2022, 2:01pm

Hi, Kyle!

I created a PR with my proposal about streaming support.

I will be grateful for the review.

PS
Unfortunately, this proposal does not solve @lastmjs problem with continuous streaming, but it allows you to download assets larger than 2mb through service-worker

cryptoschindler · February 12, 2022, 3:41pm

very cool, this is going to be super helpful! thank you for taking the time to fix this

Topic		Replies	Views
Problems loading source maps, and images referenced in `manifest.json` Developers	9	629	September 30, 2022
The difference between raw.ic0.app and ic0.app Developers Boundary-nodes	4	1156	June 7, 2022
Is there a way to get the raw url to work with custom domain? Developers	4	359	July 12, 2023
CORS error on ic0.app, failed to fetch Developers	0	322	August 29, 2022
Service worker: The script has an unsupported MIME type ('text/html') Developers	6	9111	July 17, 2021

Is it possible to make raw.ic0.app calls from ic0.app and reverse?

Related Topics