Just posting this here as documentation if it’s helpful. There should be better documentation on how to get http chunk streaming to work, since there’s a 2MB (1.9MB to be safe including headers) chunk limit on http requests and canister calls.
This example creates an output of firstchunk__middlechunk__middlechunk__middlechunk__lastchunk
https://m7sm4-2iaaa-aaaab-qabra-cai.raw.ic0.app/?tag=3614308521
if you go to:
http://raw.ic0.app/?canisterId=
Example code is also here:
Happy coding!
What’s your experience with its performance?
From my testing locally with my own Motoko http_request and http_request_streaming_callback… it’s slow. My frontend tries to fetch ~20 1-2 MB photos all at once from a single canister, and it takes more than a couple seconds to fetch them all.
If that’s the case, I don’t think canisters are gonna be performant enough to act as CDNs for photos / videos, and I might need to turn to IPFS.
It’s not fast, although I haven’t measured performance. I agree that IPFS is for sure faster.
Well damn I wonder why IPFS is faster (if it truly is)…
Not sure if the speed bottleneck is due to Motoko, or due to the http_request functions, or due to single-threaded canisters… maybe I’m doing something wrong.
Good question. I will ping team to see if anyone knows.
Wanna compare?
The following is a jpg 2.3mb image I uploaded and serve from the “asset” canister I implemented by myself: https://s2cv6-gaaaa-aaaai-aa5tq-cai.raw.ic0.app/images/2-3mb.jpg?token=c2ffb294-0413-45fa-b36e-492a705fd36e
Without cache, fastest download I got was once 1.5s and around 2.5s but, slowest is 6s. Often get something around 4s.
Are you sure those method declared as queries are actually being called as queries? For example, ‘dfx canister call’ vs ‘dfx canister call --query’. With dfx it’s possible to call a query method with much slower full consensus requirements and that’s even the default IIRC.
I think in this case the image is being fetched directly through HTTP, where the boundary node translates the GET request to a query call.
It does seem rather slow. Clicking on the link took more than 4 seconds initially and at least 2 seconds on further clicks.
For comparison, I uploaded the exact same image to Google Firebase Storage (link: https://firebasestorage.googleapis.com/v0/b/deckdeckgo-studio-prod.appspot.com/o/TLsaIXLLIwdmA2PSV1DcEHGMva23%2Fassets%2Fimages%2F2-3mb.jpeg?alt=media&token=0b071105-7bc7-4b83-add9-8f5991049c9c).
In such conventional CDN it takes around 800ms - 1s to download the file.
I get 2.5s to 6s for my implementation with the IC and http_request + streaming.
So, I actually see it as positive kind of, sure it’s slower but it’s quite different tech. Being said, maybe it is possible to improve the speed of the delivery?
Hi @spencer! I’d like to derail the conversation from speed and ask you about your implementation in the motoko_httpstream_test code you posted (thanks for sharing your work!). I was able to get the test running, but unfortunately it’s not passing. I’d love to get a chunking solution up and running so I can serve arbitrarily large videos from my canisters. Slow response speed is ok for my purposes.
To get the code to run, I had to revert it to your initial commit as your second commit deleted a some of the necessary files (Types.mo, Containers.mo…)
I was able to run the test_storage.sh script and it successfully uploaded the data to the canisters. I can retrieve one chunk at a time by making calls like:
dfx canister call storage getFileChunk "(\"testfile.txt16\", 1:nat, principal \"rkp4c-7iaaa-aaaaa-aaaca-cai\")"
and varying the 1:nat part to 2:nat, 3:nat etc for each chunk. Awesome!
But I haven’t been able to retrieve the whole file all at once. When I request the same file via url:
curl "http://127.0.0.1:8000/storage?fileId=testfile.txt16&canisterId=rkp4c-7iaaa-aaaaa-aaaca-cai"
I only receive the first chunk. streamingCallback is never called.
I’ve never worked with http chunking before, so maybe I don’t understand what’s supposed to be going on here, but I figured something on DFINITY’s end is supposed to call the callback until it returns null and send the data to my browser one chunk at a time. I shouldn’t have to write a frontend that requests each chunk and assembles them myself, like how the can-can source does it, right? Do you know why streamingCallback isn’t being called and I’m only getting the first chunk back?
Another issue I ran into when debugging the script is in updating the canister code. When I run dfx deploy (or dfx canister install --all --mode=reinstall), it seems to not update the code for the canisters that are storing the data. If I modify a Debug.print statement and then reinstall the canisters, when I request the file I don’t see my modifications. They are stuck at whatever was in the code when I ran the test_storage.sh script. I assume this is because new canisters are dynamically created for storing data, but the code for all those created canisters isn’t updated when I update the main canisters’ code. Maybe this is intentional and I’m supposed to manually update the code for each of the growing number of created canisters every time I make an update. Does that sound right?
Thanks for taking the time to consider my questions, and thanks for writing the example code in the first place! It certainly helped me get going toward my goal of hosting video files on the IC.
Hi Mike!
These are both issues I’ve encountered.
As for upgrading the child canisters, there’s no way to do that as motoko doesn’t have the necessary hooks into the ic api.
As per This.
The approach I was going to take was to use ic-repl making a script and calling an ic-repl script to upgrade each of the canisters, from a separate was generated with just the bucket code. I haven’t worked on this project in 2 months, but I’m hoping to get back on that soon. I got stuck there as I wasn’t able to get . Ic repl to work.
As for getting all the chunks to download, I’ve ran into a few issues. Make sure the url is raw.ic0.app
However I saw a response to one of my earlier posts, and I found a bug with my code as well. when I found out about this so maybe once I fix that the chunkimg will work
Thanks for your response and contribution! I’ve been stuck on that for a while!
Awesome, I’d love to follow the progress of this, and will help out where I can! For now, I’m just hosting my large files on a separate CDN, but I’d love to get the videos hosted on the IC as soon as it’s feasible.
Check out what the OpenChat team has so graciously shared on how they manage upgrading child canisters. Might be useful.
I’m also going to need a solution to store a potentially unbounded number of photos on the IC quite soon. (Would prefer to go 100% IC instead of depending on IPFS.)
