Hello everyone, Nathan here from the Trust team 
I hope you are all having a great day building on the Internet Computer 
Today I’d like to introduce you all to a new HTTP Gateway streaming protocol.
TLDR;
The HTTP Gateway enables traditional web clients to interact with canisters using the standard HTTP protocol. Due to the IC’s 2MB response size limit, a streaming protocol is necessary to handle larger responses by breaking them into chunks. The current streaming protocol has memory and security issues so we are proposing a new streaming protocol based on certified HTTP range requests. This new approach would allow for independent verification of chunks, reduce memory overhead, and enhance security, while being backward-compatible with existing canisters. Please see the details below and as always your feedback is appreciated.
Background
What is the HTTP Gateway?
The HTTP Gateway is a necessary component of the ICP network that allows traditional HTTP clients, such as a web browser, to communicate with canisters using standard HTTP protocols. If you’d like to know more about what the HTTP Gateway is, there are a number of public resources available that you can use to learn more about it:
- Smart contracts serve the web
- Boundary Nodes - Internet Computer Wiki
- HTTP Gateways - Docs
- HTTP Gateways - Spec
Why does the HTTP Gateway need a streaming protocol?
The ICP network has a number of imposed resource limits, including a maximum response size of 2mb in replicated execution mode. If a response is more than 2mb in size, what do we do? We have to break the response down into multiple smaller pieces and retrieve them one by one. Unfortunately, this is not something that a web browser will understand (at least not yet). Web browsers will expect to receive a complete response when a resource is requested, not a partial response. The HTTP Gateway uses a streaming protocol to translate multiple, partial responses into a single, complete response that web browsers can understand.
How does the streaming protocol work?
The current streaming callback protocol works as follows:
- A web browser makes an HTTP request to an HTTP Gateway.
- The HTTP Gateway converts the HTTP request into an ICP API call to the http_request method of the target canister.
- The canister responds with the first chunk, a token and the name of a canister method to call for the next chunk.
- The HTTP Gateway makes another request to the named canister method, including the provided token.
- The canister now responds with the second chunk and a second token.
- The HTTP Gateway repeats this process until the canister returns the final chunk, without a token.
- Verify the entire response and return to the client if verification succeeds.
Why do we need a new streaming protocol?
Memory issues
Almost one year ago DFINITY proposed deprecating the Service Worker on the DFINITY hosted HTTP Gateways and replacing it with ICX Proxy. The community agreed with this proposal and the work was completed three months later. Previously, ICX Proxy was only used for “raw” URLs and it did not verify streamed responses at all. As part of the transition from the Service Worker to ICX Proxy, a temporary protocol for limited verification of streamed response was introduced.
An issue with the design of Response Verification v1 is that it only allowed certifying a single response body per request URL, which meant that we were only able to certify the entire response body and not the individual chunks. By extension, this means that the entire response body must be streamed before the HTTP Gateway can verify it. When running an HTTP Gateway, such as ICX Proxy on the boundary nodes, holding all response bodies in memory can introduce a denial of service (DOS) vulnerability to the HTTP Gateway. Especially considering that the HTTP Gateway cannot control how large the response body is, how many chunks it contains, or how many users are requesting that file at the same time.
The temporary streaming protocol introduced verification of streamed responses to ICX Proxy, but only for responses that were 8mb or less in size, covering most HTTP requests made to the ICP network. This was an improvement over using ICX Proxy through “raw” without any verification of streamed responses, but a downgrade over using the Service Worker which was able to fully verify streamed responses, regardless of their size.
This temporary protocol works as follows:
- Follow the standard streaming protocol until a maximum of four chunks has been retrieved.
- If there are more chunks available, meaning the asset is greater than 8mb in size, skip verification and continue streaming without verifying any chunks / responses.
- If all chunks have been received, verify the full response, returning it to the client if verification succeeds.
Steaming callback tokens
To ensure that the correct chunk is returned on each callback, the token must include any request information that would alter the response, such as:
- The path/request URL of the asset.
- The index of the next chunk, or the index of the previously received chunk.
- Content type.
- Content encoding.
- Content language.
- Or any other form of content negotiation.
Due to the difficulty in predicting how a developer written canister will behave, the decision of what information is included in the streaming callback token is left up to the developer. This leaves the burden of token design on the developer, requiring them to understand what is necessary to include in token design and then implement it themselves. This introduces unnecessary complexity and novelty to the streaming protocol, especially considering that HTTP request and response headers already solve this problem in a way that’s familiar to web developers.
The New Streaming Protocol
Introduction
The new streaming protocol is based on certified HTTP range requests, aligning with an existing HTTP standard approach to splitting responses into multiple pieces. This protocol allows each individual chunk to be certified in the context of a complete 206 partial content response and verified independently from other chunks, removing the necessity for verification of the entire response, and by extension, removing the need to hold multiple chunks in memory. Note that this approach does not fully support HTTP range requests, as Range headers are quite dynamic and difficult to certify ahead of time. The work done for the new streaming protocol however, will support any future iterations aimed at fully supporting HTTP range requests.
Flow
When the HTTP Gateway receives a 206 partial content response for a request, the response is verified. If the response is valid, a connection is opened to the client and the response body is sent down the connection as the first chunk of bytes. Further byte ranges are requested, verified and sent to the browser, one by one, until all byte ranges have been processed. Since chunks are verified independently before being sent to the browser, the possibility of sending malicious data to the client is eliminated. Once all chunks have been processed, the connection to the browser is closed. Done in this way, the browser sees a normal 200 response and does not know that there are range requests being made behind the scenes.
- A web browser makes an HTTP request to the HTTP Gateway.
- The HTTP Gateway translates the HTTP request into an ICP request.
- If the response fits is smaller than 2MB the canister responds with a “200 OK” response, otherwise the canister responds with a “206 partial content” response.
- The HTTP Gateway verifies the response.
- If the response is a valid, complete “200 OK” response, the HTTP gateway passes the response back to the browser, and we’re done.
- If the response indicates that the canisters wishes to use the current token based streaming strategy, the HTTP Gateway complies as it does currently.
- Otherwise, if the response is a valid “206 Partial content” response:
- The HTTP gateway creates a “200 OK” response stream to the browser with the corresponding response headers and sends the first verified chunk down the stream.
- The HTTP Gateway will continue to request more chunks from the canister using range requests, verifying the received responses and pushing them down the stream to the browser.
- The stream is closed once all chunks have been received, verified and pushed to the browser.
Transition Period
This approach is backwards-compatible, meaning that existing canisters will not break when it is introduced. The HTTP gateway will work with both legacy canisters that use the token based streaming protocol, as well as with new canisters that use the HTTP range request protocol. That is, upon the first response received from a canister, the HTTP Gateway can distinguish between the following cases:
- A “200 OK“ with a complete asset.
- A partial response with only the first chunk of an asset:
- Using the token based streaming approach.
- Using the HTTP range request based streaming approach.
In any case the gateway can proceed accordingly, without additional information. There is no need for API changes, version bumps, or additional non-standard HTTP headers. In the meantime, we are also working on an asset certification library in Rust that will help canisters support this new streaming strategy, eventually including the DFX asset canister, and we are also working with the community to support HTTP certification in Motoko too. It will take time for this to be widely supported though so we expect the current streaming strategy to continue being used for quite some time.
Alternatives Considered
While looking for a new streaming solution for the HTTP Gateway, we considered two alternatives, each with their own pros and cons.
One common consideration for both of these options is the fact that they would make much smaller changes to the existing protocol. This makes the changes easier to implement and makes the transition easier for community developers, but it means that we would continue using an ICP specific streaming protocol. We believe that aligning with HTTP standards is the better approach in the long run, despite the greater short term efforts to achieve that.
Incremental computation of payload hash
The first alternative considered involved maintaining the exact same protocol that we use today, but only retain an incrementally computed hash state in memory, instead of the entire response body. This approach is simple to implement in the HTTP Gateway and does not require any changes to canisters, but it is not secure.
Since the HTTP Gateway would deliver the intermediate chunks to the browser before verifying the entire response body, potentially malicious content would be delivered to the browser. Browsers start eagerly rendering content as soon as it is received, so malicious replicas could leverage this fact to expose a remote code execution vulnerability (RCE).
New ICP Specific streaming protocol
The second alternative considered involved creating a new iteration of the current streaming protocol. This option is somewhat in-between the chosen approach and the first alternative. It aims at providing full security by verifying each chunk individually before it is delivered to the browser while minimizing the changes in the API and the need for transition mechanisms. This approach is advantageous because it would minimize developer facing changes to the protocol, reducing the impact of any transition periods.
The level of effort for implementing this iteration would’ve been similar to the HTTP aligned approach so we ultimately decided not to go with this approach and to refocus our efforts on aligning ICP technology with existing HTTP standards as much as possible.
Wrapping up
Thanks to those of you who took the time to read through this. We’d love to hear your thoughts or suggestions and are happy to answer any questions that you might have 
Happy building 