Enable canisters to make HTTP(S) requests

qijing_yu · November 7, 2022, 6:31pm

Updates to exchange_rate sample dapps (rust and motoko) in accordance with transform context change have been merged to examples repo.

apotheosis · January 16, 2023, 5:12am

@Severin @qijing_yu

Has a timeout been added to this from the IC?
I see the following error if my server takes more than 5 seconds to respond to a request.
I am fairly sure that I do not have any timeouts set on my end. Does the IC have one setup?

If so can we extent it to 60 seconds or turn it off?

“The http_request resulted into error. RejectionCode: SysFatal, Error: Timeout expired”

apotheosis · January 16, 2023, 6:17am

I found the docs about this…

the timeout period.

What is this and I think it needs to made higher…
If I put a timer on a result of 4 seconds after the first call it throws this error.

call 1: respond within 2 seconds.
call 2 though 8 : wait 4 seconds and then respond normally.

If I change the sleep to 2 seconds it works.

github.com

dfinity/portal/blob/b3178027704f8391858ca72aeedb8c110ffa1d20/docs/developer-docs/integrations/http_requests/http_requests-how-it-works.md

# Technology Overview — How Canister HTTPS Outcalls Work

On this page we provide details on how canister HTTPS outcalls, or canister HTTPS requests, work and important aspects to consider when using the API. We also want to note that there are some limitations and differences  compared to regular (Web 2.0) computer programs making HTTP calls and considerations for programmers for successfully using this feature. Engineers who intend to use this feature are advised to read through this page to get up to speed quickly w.r.t. the feature. The impatient reader who wants to dive into coding right away can skip the conceptual part and jump right away to the [coding section](#coding-https-outcalls) to get started.

## Technology

The HTTPS outcalls feature allows canisters to make outgoing HTTP calls to conventional Web 2.0 HTTP servers. The response of the request can be safely used in computations of the canister, without the risk of state divergence between the replicas of the subnet.

### How an HTTPS Outcall is Processed by the IC

The canister HTTP outcalls feature is implemented as part of the Internet Computer replica and is exposed as an API of the management canister. We next outline, in a simplified form, how a request made by a canister is processed. The HTTP request functionality is realized at the level of subnets, i.e., each subnet handles canister HTTP requests of its canisters independently of other subnets and HTTP requests are never routed to other subnets for execution.
* A canister makes an outgoing HTTP request by calling the management canister API using the `http_request` method.
* The request is stored temporarily in the replicated state of the subnet.
* Periodically (each round) an *adapter* at the networking layer in each replica fetches the pending HTTP outcalls from replicated state. (In fact, a &lsquo;shim&rsquo; layer of the adapter that is inside the replica process does so as the adapter itself is sandboxed into a separate OS-level process for security reasons.)
* The adapter on each replica executes the HTTP request by sending it to the target server.
* The corresponding HTTP response from the server is received by the adapter on each replica of the subnet and provided to a component in the replica process. The adapter limits the network response size to `max_response_bytes` which defaults to $2$ MB and can be set to lower values. It is important to set this as low as reasonably possible for the expected response as it affects the price of the request: The price increases with the size `max_response_bytes` and the actual response size is not considered, only the maximum.
* An optional transformation function implemented as part of the canister is invoked on the respective response on each replica to transform the response. This is done by a component within the replica process.
* The transformed response is handed over to consensus on each replica.
* IC consensus agrees on a response if at least $2/3$ (to be more precise, $2f+1$, where $f$ is the number of faulty replicas tolerated by the protocol) of the replicas have the same response for the request as input. In this case, consensus provides this response back to the management canister API, or an error if no consensus can be reached or in case of other problems.
* The management canister API provides the response or error back to the calling canister.

This file has been truncated. show original

tim1 · January 17, 2023, 7:38am

@apotheosis

We do have a timeout of 3s for outstanding HTTP requests. This limit has not changed since we launched the feature and I agree it might be too low for certain applications.

We will discuss raising the request timeout and I will come back to you with an answer.

tim1 · January 17, 2023, 2:49pm

Good news. We decided to bump the timeout to 30s. If everything goes well it will be included in this week’s release candidate and rolled out next week!

cyberowl · March 22, 2023, 4:41am

Is there any update if http outcalls support IPv4. Like mentioned above some services only support IPv4. One I am currently working with is the rew relic logs api. Just wondering if there is an update, otherwise I have a work around this as well. Also the error I got was Error: Failet to do wallet call. Caused by: Failet to do wallet call. An error happened during the call: 4: Failed to connect: error trying to connect: tcp connect error: Network is unreachable (os error 101)

Severin · March 22, 2023, 8:35am

I forwarded your question to the team.

Your error looks to be something with reaching the IC itself, not related to outcalls. Did you maybe forget to run dfx start or to select the right network (--network ic)?

yotam · March 22, 2023, 8:41am

HTTPS outcalls to IPv4 destinations are currently not possible, and unfortunately, we have no clear vision as to when this will be possible. The core reason is that all the solutions we discussed were decided to be not secure enough. We are now discussing other solutions (for example, that node providers will provide NAT services), but that might be a long process until we have something working on mainnet.

janosroden · April 7, 2023, 4:12pm

Hi @yotam, forming a more expensive dual stack subnet would give the same guarantees I believe

yotam · April 11, 2023, 8:53am

Hi @janosroden , I answered in the other thread now. Sorry for my late reply.

devabcd · June 1, 2023, 10:27pm

Are there any HTTP out POST examples?
For TLS auth, can a self-signed cert be used?

yotam · June 2, 2023, 7:37am

I am not aware of existing public examples of HTTP POST with this feature. To be able to make a POST request, the remote web server should support some sort of idempotency deduplication of requests. For example using an idempotency-key header.

A self-signed certificate cannot be used. The target URL must have a valid certificate.

diegop · June 2, 2023, 6:08pm

Fwiw, I am actively working on creating a code walk through tutorial with sample code for POST. I hope to be done in a few days (just need to wrap up the GET one first).

See my working draft: Http outcalls tutorial by dprats · Pull Request #1505 · dfinity/portal · GitHub

diegop · June 12, 2023, 10:29pm

Follow up:

Here is the firs part on my rewriting of the HTTPS Outcalls documentation with sample code:

This includes a walk through of the sample code in Motoko and Rust for making GET requests.

The POST version is coming soon,

ilbert · June 23, 2023, 8:28pm

As I also mentioned in this response, I’m getting a timeout error (which is not related to the transformation function, see the mentioned response for details) when running in the local replica.
What could be the problem?

diegop · June 23, 2023, 10:43pm

very odd, let me ping folks.

diegop · June 23, 2023, 10:44pm

Follow up, i forgot to share this: https://internetcomputer.org/docs/current/developer-docs/integrations/https-outcalls/https-outcalls-post

ilbert · June 29, 2023, 6:55am

Is there any update?

Topic		Replies	Views
Canister http request? Developers	1	561	June 25, 2022
Why will the IC be able to call external APIs while other blockchains can't do that? General	5	974	January 30, 2022
Make external (non-IC) HTTP request from backend canister? Programs & Applications	5	4702	May 7, 2021
Registering Web2 Webhooks Motoko	1	40	September 30, 2024
Non replicated HTTPS outcalls Developers	20	1319	August 30, 2024

Enable canisters to make HTTP(S) requests

Related topics