# Http streaming chunks in JavaScript

**URL:** <https://forum.dfinity.org/t/http-streaming-chunks-in-javascript/9746>\
**Category:** JavaScript\
**Created:** [December 17, 2021, 3:47am UTC](https://forum.dfinity.org/t/http-streaming-chunks-in-javascript/9746 "2021-12-17T03:47:23Z")\
**Posts on this page:** 12\
**Page:** 1

<div class="post-metadata">

**Author:** ![nullraisins](https://sea1.discourse-cdn.com/flex023/user_avatar/forum.dfinity.org/nullraisins/32/4026_2.png) [@nullraisins](https://forum.dfinity.org/u/nullraisins)\
**Post date:** [December 17, 2021, 3:47am UTC](https://forum.dfinity.org/t/http-streaming-chunks-in-javascript/9746/1 "2021-12-17T03:47:23Z")

</div>

Was trying to call the getAssets method of the ICPets canister in JS and am getting the following error:

`IC0504: Canister unssi-hiaaa-aaaah-qcmya-cai violated contract: ic0.msg_reply_data_append: application payload size (276532474) cannot be larger than 3145728`

Seeing the same error when viewing the method here: [Principal unssi-hiaaa-aaaah-qcmya-cai | ic.rocks](https://ic.rocks/principal/unssi-hiaaa-aaaah-qcmya-cai)

I believe the solution is to use HTTP streaming to chunk the data as discussed in this thread: [Simplest example of http streaming chunks in Motoko](https://forum.dfinity.org/t/simplest-example-of-http-streaming-chunks-in-motoko/9116). However, according to [this code comment](https://github.com/dfinity/agent-js/blob/main/apps/sw-cert/src/sw/http_request.ts#L168) it doesn’t appear that functionality has been added to the the `@definity/agent` package yet.

Am I missing something obvious or is it simply not possible to call that method from JavaScript at the moment?

---

<div class="post-metadata">

**Author:** ![diegop](https://sea1.discourse-cdn.com/flex023/user_avatar/forum.dfinity.org/diegop/32/569_2.png) [@diegop](https://forum.dfinity.org/u/diegop)\
**Post date:** [December 17, 2021, 4:27am UTC](https://forum.dfinity.org/t/http-streaming-chunks-in-javascript/9746/2 "2021-12-17T04:27:05Z")

</div>

Welcome to the dev forum, @nullraisins . Not sure myself. Let me ping to see if someone in dfinity team knows. (someone in the community may also know)

---

<div class="post-metadata">

**Author:** ![cryptoschindler](https://sea1.discourse-cdn.com/flex023/user_avatar/forum.dfinity.org/cryptoschindler/32/26_2.png) [@cryptoschindler](https://forum.dfinity.org/u/cryptoschindler)\
**Post date:** [December 17, 2021, 4:50pm UTC](https://forum.dfinity.org/t/http-streaming-chunks-in-javascript/9746/3 "2021-12-17T16:50:55Z")

</div>

I was about to ask the same question. Maybe @anon74414410 can help

---

<div class="post-metadata">

**Author:** ![anon74414410](https://avatars.discourse-cdn.com/v4/letter/a/85f322/32.png) [@anon74414410](https://forum.dfinity.org/u/anon74414410)\
**Post date:** [December 17, 2021, 6:36pm UTC](https://forum.dfinity.org/t/http-streaming-chunks-in-javascript/9746/4 "2021-12-17T18:36:01Z")

</div>

Agent-js can download chunks, but it’s the responsibility of the canister to determine when and how the chunks are returned. We don’t have a standard interface for the canister & agent to recognize that a payload is too large, and automatically retry with chunking, although that would be a nice feature.

In the meantime, the canister needs to offer chunks, or simply to expose the asset through a chunkable http query. In that case, you can fetch the asset using `fetch` or traditional `http` handling, and it doesn’t need to go through `agent-js` at all

---

<div class="post-metadata">

**Author:** ![cryptoschindler](https://sea1.discourse-cdn.com/flex023/user_avatar/forum.dfinity.org/cryptoschindler/32/26_2.png) [@cryptoschindler](https://forum.dfinity.org/u/cryptoschindler)\
**Post date:** [December 19, 2021, 2:06pm UTC](https://forum.dfinity.org/t/http-streaming-chunks-in-javascript/9746/5 "2021-12-19T14:06:03Z")

</div>

Can you maybe record a video on this? I’m having a hard time figuring out what the different ways of retrieving data from a canister look like. So far I have this understanding regarding retrieving data that doesn’t fit into a single message, please correct me if I’m wrong.

**Retrieving assets from an asset canister**  
To my understanding, when you request an asset on an asset canister via URL, this HTTP request reaches a boundary node first.

If accessed through `ic0.app`, a service worker will be installed in your browser. The service worker calls the HTTP interface of the canister directly issuing query calls to the `http_request` of the asset canister. The services worker then _should_ handle streaming for you, unfortunately it’s not [implemented yet](https://forum.dfinity.org/t/service-worker-bug-body-does-not-pass-verification/7673/5). The boundary node just forwards this call to a canister, no transformation of the request is taking place.

If accessed through `raw.ic0.app` the boundary node translates the [request into Candid which is then forwarded to the canister.](https://forum.dfinity.org/t/clearing-up-confusion-over-dynamically-storing-serving-large-assets/6764) If the asset is big enough to require streaming, the boundary node will take care of that for you. So you do not need to implement a streaming strategy in the Frontend, this happens automagically.

When using the agent-js, you’re basically in the same situation as with the service worker, no support for streaming out of the box yet.

> [@anon74414410](#):
>
> Agent-js can download chunks, but it’s the responsibility of the canister to determine when and how the chunks are returned.

^ can you give an example of how that would work?

---

<div class="post-metadata">

**Author:** ![cryptoschindler](https://sea1.discourse-cdn.com/flex023/user_avatar/forum.dfinity.org/cryptoschindler/32/26_2.png) [@cryptoschindler](https://forum.dfinity.org/u/cryptoschindler)\
**Post date:** [December 19, 2021, 2:17pm UTC](https://forum.dfinity.org/t/http-streaming-chunks-in-javascript/9746/6 "2021-12-19T14:17:12Z")

</div>

**Calling a canister method that returns a message bigger than the the message limit**  
There is no out of the box solution for that, developer need to implement their own chunking/streaming.

---

<div class="post-metadata">

**Author:** ![anon74414410](https://avatars.discourse-cdn.com/v4/letter/a/85f322/32.png) [@anon74414410](https://forum.dfinity.org/u/anon74414410)\
**Post date:** [December 20, 2021, 4:54pm UTC](https://forum.dfinity.org/t/http-streaming-chunks-in-javascript/9746/7 "2021-12-20T16:54:00Z")

</div>

I’m personally about to take some PTO for the holidays, so I won’t have a demo ready on this until next year. I will prioritize it in our first sprint of Q1 though

---

<div class="post-metadata">

**Author:** ![cryptoschindler](https://sea1.discourse-cdn.com/flex023/user_avatar/forum.dfinity.org/cryptoschindler/32/26_2.png) [@cryptoschindler](https://forum.dfinity.org/u/cryptoschindler)\
**Post date:** [February 10, 2022, 4:46pm UTC](https://forum.dfinity.org/t/http-streaming-chunks-in-javascript/9746/8 "2022-02-10T16:46:05Z")

</div>

any news on this @anon74414410 🙂

---

<div class="post-metadata">

**Author:** ![anon74414410](https://avatars.discourse-cdn.com/v4/letter/a/85f322/32.png) [@anon74414410](https://forum.dfinity.org/u/anon74414410)\
**Post date:** [February 10, 2022, 5:51pm UTC](https://forum.dfinity.org/t/http-streaming-chunks-in-javascript/9746/9 "2022-02-10T17:51:03Z")

</div>

Not yet - I’ve been tied up in the Invoice canister so far

---

<div class="post-metadata">

**Author:** ![alexeychirkov](https://sea1.discourse-cdn.com/flex023/user_avatar/forum.dfinity.org/alexeychirkov/32/1991_2.png) [@alexeychirkov](https://forum.dfinity.org/u/alexeychirkov)\
**Post date:** [September 6, 2022, 1:18pm UTC](https://forum.dfinity.org/t/http-streaming-chunks-in-javascript/9746/10 "2022-09-06T13:18:12Z")

</div>

Dear @anon74414410, do you have any news on demo? Thanks

---

<div class="post-metadata">

**Author:** ![anon74414410](https://avatars.discourse-cdn.com/v4/letter/a/85f322/32.png) [@anon74414410](https://forum.dfinity.org/u/anon74414410)\
**Post date:** [September 6, 2022, 2:57pm UTC](https://forum.dfinity.org/t/http-streaming-chunks-in-javascript/9746/11 "2022-09-06T14:57:56Z")

</div>

So yeah, updates on Http Streaming! Http Streaming has been added to the serviceworker, and now works on `ic0.app` for large files by default. We also have a bounty for an AssetManagement interface in `agent-js` that is being worked on to make it easier to upload and download files from an asset canister

---

<div class="post-metadata">

**Author:** ![domwoe](https://sea1.discourse-cdn.com/flex023/user_avatar/forum.dfinity.org/domwoe/32/5165_2.png) [@domwoe](https://forum.dfinity.org/u/domwoe)\
**Post date:** [September 6, 2022, 3:06pm UTC](https://forum.dfinity.org/t/http-streaming-chunks-in-javascript/9746/12 "2022-09-06T15:06:24Z")

</div>

> [@anon74414410](#):
>
> We also have a bounty for an AssetManagement interface in `agent-js` that is being worked on to make it easier to upload and download files from an asset canister

A bit more info on the bounty and the WIP is here: [BNT-1: Chunked upload/download library · Issue #1 · dfinity/grant-rfps · GitHub](https://github.com/dfinity/grant-rfps/issues/1)  
CC: @sea-snake
