Is it possible to make raw.ic0.app calls from ic0.app and reverse?

3cL1p5e7 · October 29, 2021, 7:21pm

I want to make a request from my application to another application via an unsafe API (ic0.raw.app), but I get an error. Is this the target behavior?

It seems that the Service worker did not resolve a canister ID from another domain URL.
Steps to reproduce:

Go to https://***.ic0.app
make fetch('https://***.raw.ic0.app/index.html')
Got error URL "https://***.raw.ic0.app/" did not resolve to a canister ID.

After debugging I found, that service worker unable to resolve canisterId when page hostname does not match requesting URL. Difference is between *.ic0.app and *.raw.ic0.app.

It’s blocker for me because I want to check certificated assets from raw resource before the load a resource in IFrame. And this is the last way to solve my problem after Why can’t IFrame be used with certificated URLs?

Please, help or answer

3cL1p5e7 · October 29, 2021, 7:36pm

Also, there is no way to register custom service worker on https://***.ic0.app, because any request to main domain is accompanied by bootstrap-page loading instead of a custom worker JS-source.

Will us be able to register out custom service workers in future?

nomeata · October 29, 2021, 9:09pm

I’m surprised: When accessing *.raw.ic0.app, no service worker should be involved. The point of raw is that it hits the HTTP Gateway on the boundary node and no certificate checking happen, so no reason to have a service worker involved.

3cL1p5e7 · October 30, 2021, 12:05pm

This applies to requests from the frontend, where the service worker is already installed. For example

go to any canister (for example go to OpenChat https://7e6iv-biaaa-aaaaf-aaada-cai.ic0.app/)
open the console
we make a request to the same canister in raw domain zone fetch('https://7e6iv-biaaa-aaaaf-aaada-cai.raw.ic0.app')
got error, Network tab is empty

nomeata · October 30, 2021, 12:08pm

Good analysis. That sounds like a bug in the service worker to me – it should let raw urls through.

3cL1p5e7 · October 30, 2021, 12:10pm

Roger!
Any suggestions who can help with this issue?

nomeata · October 30, 2021, 12:11pm

It should be sufficient to extend this check, to let requests to raw urls through (just like requests to unrelated domains):

github.com

dfinity/agent-js/blob/main/apps/sw-cert/src/sw/http_request.ts#L327

    
      
              } catch (e) {
                console.error('Failed to fetch response:', e);
                return new Response(`Failed to fetch response: ${e.toString()}`, { status: 500 });
              }
            }
          
          
  // Last check. IF this is not part of the same domain, then we simply let it load as is.
            // The same domain will always load using our service worker, and not the same domain
            // would load by reference. If you want security for your users at that point you
            // should use SRI to make sure the resource matches.
            if (!url.hostname.endsWith(swDomains)) {
              // todo: Do we need to check for headers and certify the content here?
              return await fetch(request);
            }
          
          
  console.error(`URL ${JSON.stringify(url.toString())} did not resolve to a canister ID.`);
            return new Response('Could not find the canister ID.', { status: 404 });
          }

3cL1p5e7 · October 30, 2021, 12:15pm

And I still could not find the sourcecode (sorry). Thank you so much for the link!
Perhaps I’ll open a pull request

3cL1p5e7 · October 30, 2021, 1:27pm

@kpeacock Check plz
Pull request

3cL1p5e7 · November 18, 2021, 6:24pm

Pull request was merged.
@bitdivine @kpeacock Thanks you so much!

kpeacock · November 18, 2021, 6:28pm

We appreciate the initiative, and thanks for bearing with us while we’re in the middle of migrating the serviceworker to a new codebase!

lastmjs · January 26, 2022, 12:01am

I just ran into this issue here: https://ic3o3-qiaaa-aaaae-qaaia-cai.ic0.app/

I am currently sharing https://ic3o3-qiaaa-aaaae-qaaia-cai.raw.ic0.app/ instead to get around this. All of the audio elements fail without .raw in the URL address bar.

3cL1p5e7 · January 31, 2022, 12:33pm

As far as I can see the audio works and it’s great!
If you want to use audio elements without .raw, try to certify uploaded files in the same way as certified-assets or cert-var. I haven’t tried it, but it should work.
Thank you for your feedback!

lastmjs · February 2, 2022, 7:48pm

I thought I responded to this, I fixed it up unregistering the service worker, I think it was out of date. Service workers often seem to have a problem staying up to date

3cL1p5e7 · February 3, 2022, 3:18pm

this happens because the audiofile is splited into chunks. When service-worker fetching chunk and trying to validate it, worker compares file hash from field sha256 with chunk hash. And of course they don’t match. When file was not chunkified, service-worker loads file at once and hashes match.

github.com

dfinity/agent-js/blob/cf407c49287d74f0e52395154925c8025eef0d5d/apps/sw-cert/src/sw/validation.ts#L73


      
              treeSha = lookup_path(['http_assets', '/index.html'], hashTree);
            }
          
            if (!treeSha) {
              // The tree returned in the certification header is wrong. Return false.
              // We don't throw here, just invalidate the request.
              console.error(`Invalid Tree in the header. Does not contain path ${JSON.stringify(path)}`);
              return false;
            }
          
            return !!treeSha && equal(sha, treeSha);
          }
          
          function equal(buf1: ArrayBuffer, buf2: ArrayBuffer): boolean {
            if (buf1.byteLength !== buf2.byteLength) {
              return false;
            }
          
            const a1 = new Uint8Array(buf1);
            const a2 = new Uint8Array(buf2);
            for (let i = 0; i < a1.length; i++) {

I plan to post a topic about it

3cL1p5e7 · February 3, 2022, 5:02pm

Or make streaming implementation on service-worker

UPDATE

kpeacock · February 3, 2022, 7:46pm

We’re a couple sprints in, and I’ve still been focused on the invoice canister. I’d still like to demonstrate chunking via agent-js, but that won’t resolve the basic issue of the serviceworker needing streaming support

3cL1p5e7 · February 3, 2022, 10:13pm

I have a similar question, not about streaming
Is it correct to use merkletree hashing algorithm while uploading chunks into certified_assets and make support of this on service-worker?

At now, not chunkified files works correctly (fetch(...), <img src...>) in the browser, but downloading chunkified files raises 500 validation error.

To fix this, I want to try:

calculate and save merklethee of chunks to sha256 field of certified_assets (by yourself)
[Service-worker improvement] make equal(sha, treeSha)-function to validate merkletree (because treeSha is a hash of chunk and merkletree is usually used for validating chunk hashes)

Ideally, this will allow to using chunkified assets at least in simple fetch or <img src=...> cases.

3cL1p5e7 · February 9, 2022, 2:01pm

Hi, Kyle!

I created a PR with my proposal about streaming support.

I will be grateful for the review.

PS
Unfortunately, this proposal does not solve @lastmjs problem with continuous streaming, but it allows you to download assets larger than 2mb through service-worker

cryptoschindler · February 12, 2022, 3:41pm

very cool, this is going to be super helpful! thank you for taking the time to fix this

Topic		Replies	Views
Is it possible to use iFrames to embed other asset canisters? Developers	12	1386	March 18, 2024
Problems loading source maps, and images referenced in `manifest.json` Developers	9	834	September 30, 2022
The difference between raw.ic0.app and ic0.app Developers Boundary-nodes	4	1312	June 7, 2022
Is there a way to get the raw url to work with custom domain? Developers	4	450	July 12, 2023
CORS error on ic0.app, failed to fetch Developers	0	395	August 29, 2022

Is it possible to make raw.ic0.app calls from ic0.app and reverse?

Related topics