Enable canisters to make HTTP(S) requests

[ccyanxyz]

Yeah, it’s possible.
In fact, we are already building it, called Omnic, starting from crosschain messaging between EVM chains.
We’ll release more info after the grant period ends.

[dieter.sommer]

Update on rollout

Since the end of last week, the feature is enabled on all IC application subnets, with the following exceptions:

• yinp6-35cfo-wgcd2-oc4ty-2kqpf-t4dul-rfk33-fsq3r-mfmua-m2ngh-jqe
• eq6en-6jqla-fbu5s-daskr-h6hx2-376n5-iqabl-qgrng-gfqmv-n3yjr-mqe
• mpubz-g52jc-grhjo-5oze5-qcj74-sex34-omprz-ivnsm-qvvhr-rfzpv-vae
• pjljw-kztyl-46ud4-ofrj6-nzkhm-3n4nt-wi3jt-ypmav-ijqkt-gjf66-uae
• brlsh-zidhj-3yy3e-6vqbz-7xnih-xeq2l-as5oc-g32c4-i5pdn-2wwof-oae
• qdvhd-os4o2-zzrdw-xrcv4-gljou-eztdp-bj326-e6jgr-tkhuc-ql6v2-yqe

The reason for those exceptions is that those subnets are either busy in terms of traffic, hold large state, or many canisters. We want to observe the behaviour of the feature on IC mainnet a little longer before rolling out to those subnets.

[tomkoom]

Waiting for the mpubz

[dieter.sommer]

Update on rollout

Yesterday, we have enabled the feature on the remaining application subnets.

• yinp6-35cfo-wgcd2-oc4ty-2kqpf-t4dul-rfk33-fsq3r-mfmua-m2ngh-jqe
• eq6en-6jqla-fbu5s-daskr-h6hx2-376n5-iqabl-qgrng-gfqmv-n3yjr-mqe
• mpubz-g52jc-grhjo-5oze5-qcj74-sex34-omprz-ivnsm-qvvhr-rfzpv-vae
• pjljw-kztyl-46ud4-ofrj6-nzkhm-3n4nt-wi3jt-ypmav-ijqkt-gjf66-uae
• brlsh-zidhj-3yy3e-6vqbz-7xnih-xeq2l-as5oc-g32c4-i5pdn-2wwof-oae
• qdvhd-os4o2-zzrdw-xrcv4-gljou-eztdp-bj326-e6jgr-tkhuc-ql6v2-yqe

Now the feature is available on all the application subnets of the IC.

[lastmjs]

Is the final API live on all subnets, and is the documentation and are the examples up to date with that final API?

[bitbruce]

@dieter.sommer

Customizable quorum (“unsafe mode”), where only one replica or a specified number of replicas make(s) the request, with the inherent reduction in security. This maybe useful for less critical interactions and will easily allow for making state changing requests to the outside world in the case of a single replica making the request.

This is useful. Allowing the caller to specify how many nodes to send the call as parameters would improve efficiency and success rates.

@dieter.sommer

Allowing for slightly different numerical values in responses, e.g., relevant for fast-changing data that are to be queried.

This is not necessary. Because the transformation function can achieve the same purpose.

[paulyoung]

@JaMarco

Permissioned private APIs would be really hard. A canister dev would have to get permission/access to an company’s internal APIs which would probably involve giving out a password and saving it into nodes they dont control. On Chainlink the company controls the nodes that query their private APIs.

I think this shares the same concerns as authenticating with other services, and could potentially share the same solutions as well.

The solution I proposed for authenticating with push notification services (for example) was to use a threshold key pair (as in threshold ECDSA)

The private key is split into shares so no one sees the fully assembled private key but canisters can sign outgoing requests. Only the public key is revealed in the request.

Discussing A standard for push notifications and subscriptions in wallets

I think the main concern that was brought up during Friday’s call was that at least some push notification services require the caller to include sensitive information in the request, which would be exposed to nodes in the subnet and then boundary nodes. If a bad actor were able to obtain this information and impersonate a trusted application the results could be pretty disastrous. From what I can gather, Apple (specifically) generates a key pair for you. Storing the private key in a canister …

Discussing A standard for push notifications and subscriptions in wallets

It appears that Firebase can send push notifications. Firebase projects support Google service accounts which allow uploading the public key: You can upload the public key portion of a user-managed key pair to associate it with a service account. After you upload the public key, you can use the private key from the key pair as a service account key. However: The key you upload must be an RSA public key that is wrapped in an X.509 v3 certificate and encoded in base64. I don’t think Thresh…

[singularity]

The ability to call web2 APIs directly should however be retained. You call proxy canisters for popular well known APIs if you wish to. The proxies wont keep up with all the APIs out there, and will add cost, while individual developers can.

[bitbruce]

The api.binance.com interface cannot now be called.
Error returned.

The Replica returned an error: code 4, message: “Failed to connect: error trying to connect: tcp connect error: Network is unreachable (os error 101)”

IC faces complex web2.

[Severin]

HTTPS outcalls only support IPv6 right now. I’ve seen this error happen before because a service was only available via IPv4. Could this be the problem?

[bitbruce]

Most of the interfaces in Web2 are still IPv4, I think.

[skilesare]

This is like getting your toys on Christmas morning but your parents forgot to buy batteries.

Small steps over time get us there! So excited this is rolling out. Super interested if there is a technical reason to only support v6?

[yotam]

The reason we are limited to IPv6 only is that IC nodes do not have IPv4 addresses assigned to them. This was a design decision as requiring IPv4 would have make it very difficult to scale the network.

However, we are looking at some possible solutions. None of them unfortunately is going to be perfect. Some options we are discussing are, for example, having a few subnets with IPv4, or using the boundary nodes as SOCKS/NAT64 proxies, with the possibility that the community provides more such proxies for improved decentralization.

[skilesare]

Makes sense! Thanks for the detailed explanation.

[tomkoom]

Is there a more detailed documentation how to write transform functions? I am querying https://api.coinbase.com/v2/prices/BTC-USD/spot and sometimes get an “Canister http responses were different across replicas, and no consensus was reached” error. I would only like to get the price from this query

[stefan.schneider]

My guess would be that your queries arrive at slightly different times and therefore return different prices. Is there a similar API where you can provide a timestamp?

[yotam]

@tomkoom , the reply by @stefan.schneider is correct. Maybe see the sample dapp we published that queries Coinbase Pro API through the candles endpoint, where you can specify a timestamp (could be just a few seconds into the past to guarantee consensus).

[dieter.sommer]

Breaking change upcoming

tl;dr: We intend to roll out a breaking change in the week of November 7 that will break every canister using the HTTPS Outcalls feature. Please retrofit the invocation of the HTTP Outcalls feature to match the new API to keep your canister working!

Dear community!

As mentioned by @yotam some time back, we have implemented a breaking change to the HTTPS Outcalls feature. This change seemed necessary to query some forex financial data APIs needed for the exchange rate canister that is being implemented by the Foundation. We think that everyone can benefit from the more general semantics of this and decided that the feature should be extended to support this new functionality.

Concretely, the need for the change arose because not every API out there is as nicely behaved as the standard crypto market data APIs we all know, where one can nicely query data based on time ranges and every response contains the same data, so it is easy to reach consensus on the responses with the current transformation function approach. There are services out there that don’t offer such a query interface and may return different values if calls arrive at (slightly) different times. Without this change, such APIs would essentially require a separate transformation function for each invocation, which is clearly not practical. The extension now allows essentially to parameterize the transformation function with arbitrary data, thereby essentially emulating the power of a function closure. (Note that function closures are not supported currently in Candid, so we need to make this workaround)

The change allows you to parameterize the transformation function with arbitrary data in each invocation of the features. This allows for addressing a much larger space of APIs than with the transformation function alone.

The update will be a breaking change and is planned to be voted on on Monday, November 7. Once / if voted on, the rollout to the subnets of the IC will be performed in the week of November 7 following our standard rollout schedule, where all the subnets of the IC are updated throughout the course of the week.

If you are using the feature currently, you need to change the invocation of the HTTPS Outcalls feature to contain a transformation context. The change should be small and comprise to add an empty transformation context to be compliant with the API. Your canisters will break once the change is rolled out on your subnet during the rollout week.

SDK support:

• The first SDK version supporting the change is dfx 0.12.0-beta.6, there is already a new one, dfx 0.12.0-beta.7, available now. Install with
  DFX_VERSION=0.12.0-beta.7 sh -ci "$(curl -fsSL https://smartcontracts.org/install.sh)"
• The first Rust CDK with the changes is ic-cdk v0.6.5, available at https://crates.io/crates/ic-cdk/0.6.5.

@qijing_yu has written a detailed document to explain the change and how it is performed for Motoko, Rust with the CDK and Rust without the CDK. This should also give you a hint on how to implement the chang if you are using other languages, such as C.

This is also a good opportunity to thank QJ for her work on the original release of the feature and implementing and driving the release of this feature! Thank you, @qijing_yu!

We apologize for the resulting inconvenience caused by this breaking change!

[apotheosis]

Will the example code be updated before then?

[Severin]

Example code update hasn’t hit yet, but the PR is here: #Do not merge before CDK 0.6.2 release# Updated exchange_rate sample dapps for transform with context. by qj-yu · Pull Request #357 · dfinity/examples · GitHub