Http streaming chunks in JavaScript

Was trying to call the getAssets method of the ICPets canister in JS and am getting the following error:

IC0504: Canister unssi-hiaaa-aaaah-qcmya-cai violated contract: ic0.msg_reply_data_append: application payload size (276532474) cannot be larger than 3145728

Seeing the same error when viewing the method here: Principal unssi-hiaaa-aaaah-qcmya-cai | ic.rocks

I believe the solution is to use HTTP streaming to chunk the data as discussed in this thread: Simplest example of http streaming chunks in Motoko. However, according to this code comment it doesn’t appear that functionality has been added to the the @definity/agent package yet.

Am I missing something obvious or is it simply not possible to call that method from JavaScript at the moment?

3 Likes

Welcome to the dev forum, @nullraisins . Not sure myself. Let me ping to see if someone in dfinity team knows. (someone in the community may also know)

1 Like

I was about to ask the same question. Maybe @kpeacock can help

Agent-js can download chunks, but it’s the responsibility of the canister to determine when and how the chunks are returned. We don’t have a standard interface for the canister & agent to recognize that a payload is too large, and automatically retry with chunking, although that would be a nice feature.

In the meantime, the canister needs to offer chunks, or simply to expose the asset through a chunkable http query. In that case, you can fetch the asset using fetch or traditional http handling, and it doesn’t need to go through agent-js at all

1 Like

Can you maybe record a video on this? I’m having a hard time figuring out what the different ways of retrieving data from a canister look like. So far I have this understanding regarding retrieving data that doesn’t fit into a single message, please correct me if I’m wrong.

Retrieving assets from an asset canister
To my understanding, when you request an asset on an asset canister via URL, this HTTP request reaches a boundary node first.

If accessed through ic0.app, a service worker will be installed in your browser. The service worker calls the HTTP interface of the canister directly issuing query calls to the http_request of the asset canister. The services worker then should handle streaming for you, unfortunately it’s not implemented yet. The boundary node just forwards this call to a canister, no transformation of the request is taking place.

If accessed through raw.ic0.app the boundary node translates the request into Candid which is then forwarded to the canister. If the asset is big enough to require streaming, the boundary node will take care of that for you. So you do not need to implement a streaming strategy in the Frontend, this happens automagically.

When using the agent-js, you’re basically in the same situation as with the service worker, no support for streaming out of the box yet.

^ can you give an example of how that would work?

Calling a canister method that returns a message bigger than the the message limit
There is no out of the box solution for that, developer need to implement their own chunking/streaming.

I’m personally about to take some PTO for the holidays, so I won’t have a demo ready on this until next year. I will prioritize it in our first sprint of Q1 though

3 Likes

any news on this @kpeacock :slight_smile:

Not yet - I’ve been tied up in the Invoice canister so far

1 Like

Dear @kpeacock, do you have any news on demo? Thanks

So yeah, updates on Http Streaming! Http Streaming has been added to the serviceworker, and now works on ic0.app for large files by default. We also have a bounty for an AssetManagement interface in agent-js that is being worked on to make it easier to upload and download files from an asset canister

A bit more info on the bounty and the WIP is here: BNT-1: Chunked upload/download library · Issue #1 · dfinity/grant-rfps · GitHub
CC: @sea-snake