Summary
This is a draft/preview of a motion proposal for “HTTP Requests from Canisters” which we will submit tomorrow (February 23, 2022) for the community to vote on.
Once proposal is live, I will update the forum.
Background
Canister smart contracts cannot make requests to HTTP/HTTPS services on the Internet by default. Doing so is a challenge on any blockchain caused by the fact that different replicas / nodes can (and will) receive different responses for the same call, be it, for example, for timestamps or ids contained in the response. Such different responses received by different replicas and further processing being based on those different responses on each replica would lead to a divergence of state on the different replicas and thus destroy the determinism property of the computation with the result that no consensus could be achieved. Thus, replicas cannot just make HTTP/HTTPS requests to outside services without creating a major problem for the subnet.
However, it is technically possible to enable canisters on the IC to safely make HTTP/HTTPS (henceforth just “HTTP”) requests with an extension of the Internet Computer protocol. This feature discusses such extension to the Internet Computer protocol. We think that this is a crucial step towards better integrating the Internet Computer with the public Internet and thus breaking down so-far inherent borders that blockchains face. We think that opening up the Internet Computer to interface it with HTTP services on the Internet is a major step towards the future of the Internet Computer as a platform that can run general-purpose workloads and also for a more open, integrative, blockchain world at large.
Today, obtaining data from the outside world requires, as on any blockchain, the use of blockchain oracles, or oracles. However, oracles lead to a more complicated programming model, charge substantial fees, add complexity and indirections, and require additional trust assumptions to be made. Allowing canister code to directly make HTTP requests would remove the dependence on oracles and its disadvantages. Of course, oracles may still be useful for certain use cases when we have HTTP calling capabilities, but many use cases could be covered with direct HTTP calls.
Allowing canisters to make HTTP calls to services on the public Internet has long been a community-requested feature for the Internet Computer. It will give smart contract canisters the ability to autonomously connect to services on the Internet to retrieve or submit data and will thus enable a large range of additional or enhanced use cases, e.g., obtaining exchange rate data from external servers for DeFi applications, obtaining weather data for decentralized insurance services, or sending notifications to users via traditional communications channels, all without using oracles. HTTP support for canisters is one of the features on our strategic R&D initiative on “General Integration” (see Long Term R&D: General Integration (Proposal) - #4 by dieter.sommer), thus we want to now proceed with launching a motion proposal for this feature and ask for community approval w.r.t. going forward. If accepted, a launch in the Q1 Chromium release is planned, meaning an aggressive timeline for our engineering teams.
Goals and Requirements
This feature should enable canisters to directly make requests to HTTP URLs, using the GET method initially, and receive the corresponding response back into the canister’s state in a deterministic fashion. The functionality should be realized in a direct, trustless, way. Direct, trustless, integration is a common theme in other integrations of the IC, e.g., the Bitcoin integration that is to launch on mainnet in the near future. Direct integration means that we do not need to make any additional trust assumptions or involve any additional parties to realize the functionality.
For the first version, all replicas in the subnet will send out the request and return a response that goes through the IC Consensus mechanism. In the future, we may present an option for canister developers to reduce the size of the required quorum, so that even only one replica may make the request, if desired, but the guarantees on such request would be accordingly lower. Another future envisioned extension are POST requests. In combination with the reduced quorum, those can be of tremendous utility for many use cases where reliability of the calls is less important than compatibility with APIs out there.
Proposed Design
We next outline the proposed design at a high level.
System API (Management Canister)
We implement a system API in the Management Canister that provides a method for making an HTTP/HTTPS call to an outside service and receiving back a response. See below for the original proposal w.r.t. the API and note that not all community feedback from the discussions in this forum topic have not been included here yet.
type http_header = record { 0: text; 1: text };
type http_response = record {
status: nat;
headers: vec http_header;
body: blob;
};
type http_request_error = variant {
no_consensus;
timeout;
bad_tls;
invalid_url;
transform_error;
dns_error;
unreachable;
conn_timeout;
};
New method in ic0:
http_request : (record {
url : text;
method : variant { get };
headers: vec http_header;
body : opt blob;
transform : opt variant {
function: func (http_response) -> (http_response) query
};
}) -> (variant { Ok : http_response; Err: opt http_request_error });
See the (draft) PR on the interface specification repository, which is now public, for details regarding the proposed API and related discussions: IC-530:Canister HTTP requests by ielashi · Pull Request #7 · dfinity/interface-spec · GitHub.
High-Level Request and Response Flow
Calling this API will store the request in a specific area of replicated state that is periodically read by a component at the networking / consensus layer. This component, once it sees a new request, provides this request to an HTTP Adapter at the networking layer which performs the actual request and provides a response in return.
Responses are put into a new HTTP Artifact Pool, are signed by the replica to endorse the response, and the signature is gossiped to all replicas in the subnet. Once a request has support by at least 2/3 of the replicas of the subnet in the view of the current block making replica, it adds this endorsed response to an IC block that is going through Consensus. Because at least 2/3 of the replicas of the subnet have supported the response, it is ensured that the subnet can achieve consensus on it.
Once the IC block with the HTTP response has made it through the IC consensus layer, it is routed back to the system API and is provided back to the calling canister, which concludes the original API call for making an HTTP request.
We do not go into the details of the error handling. In short, it is possible, as error scenarios, that requests time out or cannot be consented on, in which case a corresponding error response is generated and returned in response to the request.
Handling Differences in Responses
Many HTTP-based services like API providers include fine-granular timestamps or unique ids into their responses, implying that it would not be possible to achieve consensus on the responses received by the different replicas in the subnet. This can be addressed by allowing the caller to specify a response processing function to be performed on the responses before they are provided to the consensus layer. This allows for a much broader field of application for the feature by allowing a broader class of responses to obtain consensus on.
The canister may specify a response processing method that, when a response is received by each replica, is applied on the response on each replica to transform it accordingly into a response that is intended to be the same on each replica and thereby will be accepted by the IC consensus mechanism.
The transformation may, for example, only keep specific fields from the responses, while removing other values that might differ across responses, such as timestamps or unique identifiers. The transformation can also just retain a single value of interest from the whole response, e.g., an exchange rate value, which would substantially reduce the required IC “block bandwidth”.
The design choice to expose a canister method to perform the transformation and not do the transformation directly in replica code has multiple reasons behind it:
- The computational effort for the transformation can be directly accounted for through consuming the canister’s cycles. Thereby certain kinds of denial of service attacks that would be possible and would need to be addressed for a replica implementation are not possible.
- It is fully flexible in terms of which transformations can be implemented. Implementations in replica code would use a specific approach, e.g., a templating language, for defining the transformation.
A drawback of the approach of exposing a canister method for the transformation instead of an alternative considered design of allowing for a set of transformation types parameterized by a template as input to the method call is that it may be slightly more effort on the side of the canister author to implement the canister method. However, the tradeoffs have been considered substantially in favour of the approach of exposing a canister method, as in the other approach it would be difficult to charge for the transformation effort and to prevent DoS attacks using long-running transformations.
Roadmap
We plan, assuming a supportive community vote, to build the feature to be ready for a release around the end of Q1 / 2022. We propose a design and scope that is reasonable to implement for a first MVP as outlined in this motion proposal to trigger further discussions and support a community vote.
The implementation cuts through all the layers of the IC protocol stack and thus requires tight collaboration between the core IC engineering teams. Most engineering effort is expected on the consensus layer, next are networking, execution, and message routing in descending order of effort. In order to meet the tight timeline to a Q1 release, the different teams will work in parallel as far as reasonably possible.
In order to ensure the high quality of the feature, we will perform extensive automated testing in our system testing environment and a security review.
Extensions
The envisioned feature implementation is a first MVP that provides the core functionality of allowing canisters to make HTTP requests. We have already identified some enhancements that have been decided to not be implemented as part of the first release, but that we can realize as separate features in the future.
- POST/PUT requests: Those would be pretty similar in terms of implementation assuming idempotency of the requests. Not assuming this inherently requires us to use a reduced quorum of size 1 to emulate traditional POST/PUT calls or to extend the called API such that it can handle multiple requests for the same POST/PUT to be done and execute them only once.
- Customizable quorum (unsafe requests): This allows the canister to specify the quorum size to make a tradeoff between performance, resource consumption, and compatibility with traditional HTTP-based servers on the one hand and security on the other hand. The most relevant reduced quorum size in practice will be quorum size 1. This extension, together with the possibility of making POST/PUT requests will enable another large array of use cases without making changes to external services.
- Persistent connections: This is an extension purely for better performance, and thus left as an extension instead of implementing it already as part of the MVP.
- Different numerical response values: Some APIs will result in slightly different response values if called at slightly different times. The latter is typically the case in the setting of all replicas making the same call to a service. A further extension to the feature can allow for such different received numerical response values to be consented on and being returned in appropriate form, e.g., their median or all values are returned so that the calling canister can directly receive or apply an appropriate function to determine the “actual” response value.