Enable canisters to make HTTP(S) requests

Roadmap
1

Summary

Smart contracts on the Internet Computer, like on any other blockchain, cannot communicate with external servers per se, e.g., via making HTTP(S) requests. Enabling canisters to make HTTP(S) requests will help integrate the Internet Computer with the Web 2.0 world. This allows for a plurality of new use cases, e.g., obtaining exchange rate data from external servers for DeFi applications, obtaining weather data for decentralized insurance services, or sending notifications to users via traditional communications channels. The first version of this feature will cover a subset of the functionalities that are typically provided by oracle services, but will do so in a trustless manner.

Status

Discussing

Key people involved

Yotam Harchol, Ivan Malison, Islam El-Ashi, Manu Drijvers, Thomas Locher, Akhi Singhania, Alexa Smith, Dieter Sommer

Description

It is not possible for canisters currently to make HTTP(S) requests to servers outside the Internet Computer. This limits what canisters can do as it lacks an integration with the Web 2.0 world. The reason that it is not easy to let canisters make HTTP(S) requests to any external system is that this is a non-deterministic process and all replicas in a subnet need to operate in a completely deterministic manner. A subset of the HTTP(S) scenarios can be captured with an approach of every replica making the same request and then achieving consensus on the response. This is not straightforward as responses for the same call, e.g., to a price API, may differ in some parts, e.g., identifiers or timestamps. Doing a transformation on the HTTP(S) responses before feeding them into consensus can solve this problem.

According to the current thinking, the first MVP for this feature should comprise the ability to make HTTP(S) GET requests, specify a transformation to be applied to the responses by each replica, and then running consensus over the responses. This will already allow for a large set of use cases.
State-changing requests to standard services are problematic in this model as the request is executed by every replica. If the external service, or a fronting service built for the IC interaction, is aware of this model, state-changing calls can be done as well once POST/PUT calls are available.

Later extensions may comprise, among other things, the following:

  • POST/PUT requests: enables updates to the outside world if the call model of every replica making the request is accounted for.
  • Customizable quorum (“unsafe mode”), where only one replica or a specified number of replicas make(s) the request, with the inherent reduction in security. This maybe useful for less critical interactions and will easily allow for making state changing requests to the outside world in the case of a single replica making the request.
  • Allowing for slightly different numerical values in responses, e.g., relevant for fast-changing data that are to be queried.
  • Persistent connections (for better performance for periodically executed queries)

Context

Enabling canisters to make HTTP(S) requests is one of the topics of the larger long-term R&D motion proposal on General Integrations (see Long Term R&D: General Integration (Proposal)). It has also been picked up by the community in another topic in the dev part of the forum (IC-530 "Canister can make HTTP requests") after it has been “leaked” as part of an Inside DFINITY episode (see Direct Integration with Bitcoin - #70 by skilesare, Idempotent egress messages (e.g send an email) - #2 by wang) where it has already been discussed in an internal DFINITY meeting in the context of the Bitcoin integration feature because of possible architecture synergies.
Some other related discussion has already been ongoing in multiple other places in the forum.

This is a cross-cutting feature requiring work in the networking, consensus, message routing, and execution layers. Engineers from each of the teams will join the discussion here so we can answer any question that comes up.

37 Likes
2

Would love to see this up and running. Would make trustless bridges possible between the IC and many chains.

8 Likes
3

I think is this is an extremely powerful feature and key to hosting more of the worlds critical software via smart contracts

8 Likes
4

Very nice

Would it truly be trustless? From my limited understanding the symmetric encryption in TLS causes problems. Some workarounds are https://www.deco.works/ and https://tlsnotary.org/

Interested to know how this would work

3 Likes
5

Maybe an easier first step is to enable “fire-and-forget” requests where you don’t care about the response, e.g. push notifications. See this related thread.

Also, how does the BTC solve this? Apologies if this has been answered before.

3 Likes
6

Regarding what we currently have in mind for a first MVP, please see the edit of the topmost post with additional details.

The Bitcoin integration feature solves this by having a more specialized implementation targeted at Bitcoin in which each replica essentially acts like a lightweight Bitcoin client that has some state regarding the Bitcoin network, synchronizes with the Bitcoin network and feeds blocks into consensus.

The consensus integration is quite different, though:

  • Bitcoin blocks are what we call “self-validating payloads” as they can be validated, e.g., based on the hashwork that has been used to mine the block. That is, we do not need to achieve consensus of what each replica sees, but blocks can be handled much like ingress messages with some additional checks regarding the validation.
  • HTTP(S) responses, on the other hand, are non-self-validating payloads, i.e., we need to have agreement between the replicas on what the correct response is. That is, we require a more complex integration with consensus that is at the core of this feature.

Hope this answers your question, otherwise feel free to ask again. :slight_smile:

4 Likes
7

I don’t see this as a priority and would much rather see services pop up that handle this in a more secure, auditable, and deterministic manner.

This feature would never be useable by enterprise applications since the receipt of the HTTP response would never be guaranteed to reach a consensus. The handling of this seems like a nightmare inside IC code. It seems much more straightforward to have trusted applications that watch the IC, respond to events, and then put the desired response back into the IC.

Further, because most web 2 systems are not written with multiple responses coming in that need to match 100% this seems like a massive foot gun for onboarding devs that want to use this feature and just can’t get it to work because twitter has a timestamp it injects into each response that makes get requests never match.

2 Likes
8

Yeah agree if we’re just gonna be recreating chainlink.

One option I’ve been looking at is Mina: HTTPS and Snapps: Bridging cryptocurrency and the real world | by Mina Protocol | MinaProtocol | Medium

A SNARK is created that checks the https response signature and whatever is inside the body.
There’s not much details out (supposedly coming early 2022), but from what I gathered its similar to deco. i.e. you need some third party participating in the handshake (their snapp operators) or else it could be forged.

The benefit is you only need 1 request so no consensus needed, and anyone could do the request from their own PC and create a proof that checks private information (like bank account balance) without revealing that info.

1 Like
9

I think it would be great to have “Chain-Link” on the IC, the consensus mechanism would be more secure. We are really moving to a multi-chain world and this would blow most other blockchains out of the water.

4 Likes
10

Hi Max!

We have a motion proposal about “General Integration” (Long Term R&D: General Integration (Proposal)) which will comprise, among other topics, integrations with other blockchains and oracle providers. Chainlink integration, also with their upcoming Cross-Chain Interoperability Protocol (CCIP; Cross-Chain Interoperability Protocol (CCIP) | Chainlink), is something of relevance there. Also things like an integration with Polkadot as mentioned in one of the threads should be discussed as one option to integrate with existing integration layers.

1 Like
11

Let me try to clarify a little what we have in mind. From what you write it seems you are assuming that the canister making an HTTP(S) call will need to deal with all the (slightly) different responses from all replicas of the subnet. However, this will be done by the IC, the canister gets back a single response. The only additional thing the canister needs to care for is to define a transformation function that is applied to the response by each replica to “cut out” varying information, such as timestamps of request-specific ids or just return a single number of interest, e.g., a price of an asset. The IC replicas then apply this transformation on their respective HTTP(S) response and thereby we have the same consolidated / transformed response to account for variable bits and pieces in the response on each replica that will be fed to consensus. A single consolidated response will reach the canister like in a traditional Web 2.0 application. Thus, querying will work really nicely with this approach, for HTTP(S) endpoints that return predictable responses, e.g., various API endpoints. There is no need for external trusted parties and thus additional trust assumptions. It’s essentially a direct integration of oracle-like functionality into the IC protocol stack. This is hard to beat and no other blockchain would have this feature. Also request cost would be much lower than when using oracle services.

Quoting Max:

The more difficult problem is when updating state in external services. Either the service can handle the same state-changing call arriving multiple times, which is easily doable when building such a service with blockchain clients in mind, but current standard services do not account for this. Building a proxy towards external services that handle those multiple update calls is simple, but again introduces a trusted party. This requires definitely further discussion.

External services that pop up would always add additional trust assumptions and thereby make it less secure. In the ideal case, we lose little to no security, but that’s already hard to achieve in practice when introducing those additional trust assumptions.

I think that the external services you mention could coexist in the larger integration landscape and provide additional value for specific integrations. When you speak about enterprise applications, what exactly do you mean? Maybe that’s a specific use case.

7 Likes
12

Quoting myself above, let me also talk a little bit more about this as it touches the problems that @skilesare and @Tbd have voiced. The functionality of a customizable quorum would be an extension that allows the calling canister to define how many replicas should do the call, e.g., this could be 1, thereby allowing for a trade-off between security on the one side and performance, simplicity, and cycles cost on the other side. When doing a request with 1 replica only, we would give up some security which may be fine for many use cases. Also update calls to external systems would be trivial, with the same reduction in security. It may be good enough for things like many user notification scenarios etc.

3 Likes
13

Thanks for sharing those links to other integration approaches!
This looks pretty interesting after a quick glance over it. I would be interested in seeing more details on how this is actually done, maybe next year, once they have released it. :slight_smile:
However, one drawback seems to be that there is again the need for additional trusted parties that take part in the TLS handshakes.

2 Likes
14

Indeed, with the additional extra functionality we have listed, we would get close to what Chainlink can offer. Our approach would likely be massively cheaper per call, more flexible for users, and provide better security than a single oracle query. Doing multiple oracle queries in chainlink then again costs even more and puts additional burden on the canister to do a “poor man’s consensus” on the individual results on the application level. Our proposal would move most of the complexity into the IC protocol stack, at the cost of the implementation effort we have.

I greatly appreciate the discussion and alternatives that are being proposed and think that we will likely need a broader landscape of features and services to achieve all that we want to achieve. We learn more about what our community thinks is the right approach to go forward with with every blog post! Those discussions will help us in prioritizing our efforts toward maximizing the value we create.

6 Likes
15

It’s essentially a direct integration of oracle-like functionality into the IC protocol stack. This is hard to beat and no other blockchain would have this feature. Also request cost would be much lower than when using oracle services.

I wonder why other blockchains don’t do a direct integration like this. Is it just too much effort to implement? Or are there deeper technical requirements to this that are only met by the IC?

2 Likes
16

I think Chainlink also has done some work on TLS handshakes but it’s not completely trustless, still needs a consensus engine to run on top.

3 Likes
17

This TLS handshake-based approaches are definitely worthwhile looking at in more detail!

2 Likes
18

deco is the one that was acquired by chainlink :smiley: recommend checking out their paper
they also acquired https://www.town-crier.org/ Lol, but it uses secure enclaves

1 Like
19

Two potential reasons:

  • Thanks to the subnet-based architecture of the IC, only the replicas of one subnet handle an HTTP(S) request, and not all replicas of the blockchain. This is definitely a big advantage in terms of scalability.
  • Our consensus implementation is rather flexible in terms of handling different kinds of payload that require different processing. E.g., ingress messages, Xnet messages, DKG-related and soon also threshold-ECDSA-related messages and HTTP(S) messages. Not sure, whether the consensus layer of every blockchain provides that much flexibility. Any experience here?

That’s the things that come to my mind, but maybe there are more. Any other thoughts on this?

5 Likes
20

sounds great!!!
followed~
:star_struck: