Boundary node http response headers

Now everything is clear for me, thanks a lot!

Do you have any plans for this in the near future?

I don’t.

BTW, for many applications it might be a good option to serve HTML and JavaScript safely over the certified URL, and then do more complicated stuff (e.g. fetch video chunks certified by a stream signature) from your JS in an application-specific way, and not using the certified HTTP stuff in that case. This way no need to wait for me or DFINITY to find generic solutions (which tend to be harder).

2 Likes

Guys this is a humble opinion on two things that I feel are faces of the same decentralization coin.

  1. _raw
    This is a security hole I wish Dfinity as org had never introduced. The implication of a single replica serving arbitrary data that cannot be checked for integrity is difficult to digest. Also now that _raw is deployed in the wild it’s difficult to roll back such decisions.

  2. http header
    My opinion is that Dfinity should NOT provide guidance/spec out “web content related” canisters. HTTP is an independent spec, what does mean to have a restrictive/additive HTTP spec within IC spec. It’s a slippery slope why stop at webserver specs, lets’s also do mail server specs

I sympathize with @lastmjs, he is being coerced into doing streaming “a” particular way. Having said that I don’t think adding anything to spec or boundary nodes is the right idea.

Instead, let’s work towards some solution where
@lastjms can run his canister, and his boundary nodes and expose any HEADER, any un/secure content he wants to. There is dissonance in what we are asking developers to do. Providing an unsecured _raw path, having an opinion on streaming be done X vs Y way is not an IC concern.

The effort required may be similar.

2 Likes

I agree with your words!

About your suggestion: there are some limitations for boundary node:

  • The deploing own boundary node under custom domain is might be a great solution. But then it will lose the ability to work under ic0.app domain, especially when custom subdomains are released. Demergence-podcasts will live fine on demergence.com (for example), but not on potentially demergence.ic0.app.
  • This may can affect the audience’s trust (not a fact)
  • The streaming feature needs not only for demergence, it might be useful for others. And the easiest way to get streaming feature for ordinary developer is to use certified_assets canister (simple set "type"="assets" in your dfx.json and lets gooo)

In general, a solution is acceptable. Moreover, there is an info about deploy boundary node

@lastmjs
I’ve been thinking about this for the last week and can suggest the next:

  • In theory, you can to develop custom service-worker with range streaming support and register it on your canister manually. It will catch .raw. requests with range headers, will try to collect chunks from canister query methods and will return response with ranges to the client. There is my attempts to do this and it works nearly fine with TODOs for certfied_assets

To whom it may concern: I’ve been working with Daniel to try to figure out this strange safari issue where I can’t get our video to play from our canister. If I run a local version of the icx-proxy and point it at ic0.app I can get my video to play at http://localhost:3000/-/1/ex?canisterId=r5m5i-tiaaa-aaaaj-acgaq-cai but if I call the same thing at https://r5m5i-tiaaa-aaaaj-acgaq-cai.raw.ic0.app/-/1/ex the video does not play.

In both cases, safari stops the request when it sees ‘video’ in the content type and sends a new range request asking for the first two bytes. When I go through a local proxy that range request gets through(It is a header on the request (Range: bytes=0-1). When I go straight to raw.ic0.app Safari says that it is sending the header, but i’ve manipulated my server to return the actual headers that I’m getting and the range request is not there. So either Safari is lying(Daniel thinks so) or something is stripping off the range request on the way to my canister. I don’t know the topgraphy of where a request actually goes, but I suspect there is a boundary node and a proxy?

Request - through local version of icx-proxy
GET /-/1/ex HTTP/1.1
Accept: */*
Connection: Keep-Alive
Range: bytes=0-1
Host: localhost:3000
Accept-Language: en-US,en;q=0.9
User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/15.0 Safari/605.1.15
Referer: http://localhost:3000/-/1/ex?canisterId=r5m5i-tiaaa-aaaaj-acgaq-cai
Accept-Encoding: identity
X-Playback-Session-Id: 6FFA5EE3-52F6-4DDC-8297-300A4C22A9DF

Response
HTTP/1.1 206 Partial Content
Content-Range: bytes 0-1/46984888
Accept-Ranges: bytes
Content-Type: video/mp4
Content-Length: 2
Date: Thu, 21 Apr 2022 22:58:16 GMT


Request 2 - through raw.ic0.app
GET /-/1/ex
Range: bytes=0-1
Accept: */*
Referer: https://r5m5i-tiaaa-aaaaj-acgaq-cai.raw.ic0.app/-/1/ex
Accept-Encoding: identity
Connection: Keep-Alive
User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/15.0 Safari/605.1.15
X-Playback-Session-Id: 62FA3A12-FD2A-4C34-839A-B6963DCFC6F1

Actual headers that get to my canister
[("host", "r5m5i-tiaaa-aaaaj-acgaq-cai.raw.ic0.app"), 
("x-real-ip", "108.230.245.93"), 
("x-forwarded-for", "108.230.245.93"), 
("x-forwarded-proto", "https"), ("connection", "close"), 
("user-agent", "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/15.0 Safari/605.1.15"), ("accept", "text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8"),
 ("accept-language", "en-US,en;q=0.9"), 
("accept-encoding", "gzip, deflate, be")]

Response:
200
Access-Control-Allow-Credentials: true
Content-Type: video/mp4
Access-Control-Allow-Methods: GET, POST, HEAD, OPTIONS
Access-Control-Expose-Headers: Content-Length,Content-Range
Access-Control-Allow-Origin: *
Date: Thu, 21 Apr 2022 22:56:06 GMT
Access-Control-Allow-Headers: DNT,User-Agent,X-Requested-With,If-Modified-Since,Cache-Control,Content-Type,Range,Cookie
Server: nginx/1.21.3
x-cache-status: MISS

This thread confirms my suspicions.

Looks like the short term solution is to set up a server running icx-proxy that can server your requests. So @lastmjs you could set up stream.demergence.com/episode-tag?canisgerId=xxxxxxxxxxxxx and it would work.

As far as certified assets go, range requests allow you to return the results you want vs. what was requested, so if you certified 1MB chunks you could always return at least that chunk. If it requests something in the middle of a chunk, rewind to the earliest certified starting point and return the chunk. maybe the chunks need to be smaller, but since safari requests them in order and the latency is pretty bad it can take along time to download large files.

A bit irrelevant as it seems like icx-proxy does not certify data anyway (but it would be great to have confirmation at Can the icx-proxy return certified assets?

The boundary nodes are most likely removing the Range header, that’s exactly the issue I was having

Line 390

This is not the nginx configuration on the boundary nodes but its a exact copy. You can check how the boundary is configured with respect to header stripping by looking at this code.

So if it isn’t being stripped, do you think that safari is just not sending it properly?

If it’s getting stripped it doesn’t matter how safari is sending it. It will get stripped

Implementing our own version of a ICX proxy should fix this issue, correct?

@skilesare no, the next step for the issue you are facing would be

-what header is being stripped and where (in the nginx source code i shared)
-motivate inclusion of the header/or workaround the header.

Currently AFK, would love to get your problem resolved under the video streaming effort already underway at Dfinity. there are many ways to do video streaming - the core focus is to select one that doesn’t break boundary node caching

Boundary node caching is absolute must if the IC is to host YouTube scale videos. So it’s not just about simply enabling range headers. What’s your project/contact point I will try to get you plugged into the video effort

Well…I have it mostly solved for my purposes through a local icx-proxy. We’re going to deploy it to a domain early next week and see if it continues to work.

You can simulate your self by running a local icx proxy and pointing it to ic0.app and hitting that same url from safari. It does some strange things for that video about 30MB in, but full video ends up coming down.

Added you to an on going conversion.

Can we get an update from DFINITY on when boundary nodes will allow for range requests? I would also love to know the current strategy for supporting streaming audio/video and when we can expect this all to be released.

My podcast Demergence is hosted 100% on the IC but its streaming has been broken since the beginning since there is no support for range requests. I would to get this resolved soon so we can have full podcast support on the IC!

4 Likes

Range requests work through icx proxy.

See: https://nft.origyn.network/-/nftforgood_uffc/-/ogy.nftforgood_uffc.1/social

Even works in safari for the most part.

What about on a regular IC subdomain with no proxy?

Nope. They block the header for some reason.

I thought every boundary node (that every request must go through) runs icx-proxy. Can you clarify what you mean by a “regular IC subdomain”?

I think @lastmjs is referring to boundary nodes, whereas I think @skilesare is saying that Origyn is running their own instance of icx-proxy somewhere.

Yes. This is what I mean. Although I still have confusion as to if when we run icx proxy and point to ic0.app if we are ourselves using a boundary node. are we proxying to a proxy?