Hi @spencer! I’d like to derail the conversation from speed and ask you about your implementation in the motoko_httpstream_test code you posted (thanks for sharing your work!). I was able to get the test running, but unfortunately it’s not passing. I’d love to get a chunking solution up and running so I can serve arbitrarily large videos from my canisters. Slow response speed is ok for my purposes.
To get the code to run, I had to revert it to your initial commit as your second commit deleted a some of the necessary files (Types.mo, Containers.mo…)
I was able to run the test_storage.sh script and it successfully uploaded the data to the canisters. I can retrieve one chunk at a time by making calls like:
dfx canister call storage getFileChunk "(\"testfile.txt16\", 1:nat, principal \"rkp4c-7iaaa-aaaaa-aaaca-cai\")"
and varying the
1:nat part to
3:nat etc for each chunk. Awesome!
But I haven’t been able to retrieve the whole file all at once. When I request the same file via url:
I only receive the first chunk.
streamingCallback is never called.
I’ve never worked with http chunking before, so maybe I don’t understand what’s supposed to be going on here, but I figured something on DFINITY’s end is supposed to call the callback until it returns null and send the data to my browser one chunk at a time. I shouldn’t have to write a frontend that requests each chunk and assembles them myself, like how the can-can source does it, right? Do you know why
streamingCallback isn’t being called and I’m only getting the first chunk back?
Another issue I ran into when debugging the script is in updating the canister code. When I run
dfx deploy (or
dfx canister install --all --mode=reinstall), it seems to not update the code for the canisters that are storing the data. If I modify a Debug.print statement and then reinstall the canisters, when I request the file I don’t see my modifications. They are stuck at whatever was in the code when I ran the
test_storage.sh script. I assume this is because new canisters are dynamically created for storing data, but the code for all those created canisters isn’t updated when I update the main canisters’ code. Maybe this is intentional and I’m supposed to manually update the code for each of the growing number of created canisters every time I make an update. Does that sound right?
Thanks for taking the time to consider my questions, and thanks for writing the example code in the first place! It certainly helped me get going toward my goal of hosting video files on the IC.