As far as I know, it is currently impossible to host podcasts with full functionality from a canister. I know because I’ve been hosting my podcast Demergence from a canister.
Why doesn’t it work? Basically because boundary nodes have a whitelist of http headers and block custom http headers that would be required to implement range requests. Range requests are required for podcasts to seek appropriately from podcast players.
I have already implemented a temporary solution to range requests that work with a modified asset canister. I can’t deploy it though because boundary nodes strip out the needed headers.
Apparently boundary nodes are being upgraded to hopefully allow range requests natively. But this isn’t live yet as far as I know.
I’ve been waiting for a very long time for boundary nodes to not strip out the http headers that I need. What is the status on this? An entire class of hosting us impossible because of this whitelist. My understanding is that it’s just out of an abundance of caution, not because of ant specific identified security threat.
If someone from DFINITY can please give an update and if we could get this functionality out, that would be great.
Sorry for the long delay in response. I am copying @raymondk Raymond (Sr. Eng Manager) who owns this part of the stack.
We will modify BNs so they forward the range headers for the .raw endpoints, in order to unblock you and others like you. There are two things to note:
It’s possible that in future versions of certified request/response, the solution of “forward the range headers for the .raw endpoints” may have to be dropped, so there is a temporary nature to this solution. Unclear when this would happen, but if it does, any new version will be better or equal to what would be deprecated. It may require you to make changes to your canister.
In general, range queries are not good for caching so this is not an optimal solution, but we understand this is necessary for your streaming from platforms like Apple, etc.
In order to make this easier/faster for us to scope, can you share with us the canister + canister code from which you would stream your podcasts? We may also need your help testing.
If the IC is going to be a general-purpose cloud, then I don’t think we should remove an entire valid class of http functionality. Some streaming may be more optimal in certain ways, but range requests are a valid way to perform various use cases.
I say this because there seems to be a mentality inside of those in charge of BNs that range requests are some sort of invalid functionality. Canisters are going to do all sorts of things using the http standards, I hope we’ll support everything as far as possible.
Needless to say it’s my understanding that podcast clients use range requests, so there’s nothing that can be changed there unless we just don’t want to be able to host podcasts on the IC.
It basically just adds rudimentary range requests to the assets canister, it was at least working locally well enough that I was excited to deploy it into production and test it out. Did you want a canister id as well?
Thanks for sharing the repo @lastmjs! I think this is good for now.
We should be able to use your code to verify that our changes are working properly.
That pretty much was the deciding factor… it does not matter how elegant/inelegant X feature is, it matters more what it unblocks. We all agreed that allowing podcasting (and similar) is an important use case.
It’s been a while, but now that Express is working in Azle I decided to deploy an example application to mainnet to test out audio and video streaming.
It looks like the Range request header is being stripped out by the boundary nodes or somewhere along the chain, everything is working fine locally but on mainnet it looks like the range header is undefined.
Can we please get Range headers enabled, and all other headers that canisters might want to accept or return? This is a major bummer, it would be amazing to be able to stream audio and video and Azle might be doing it but is now blocked by this issue that I hoped had been resolved years ago.