Long Term R&D: General Integration (Proposal)

[diegop]

1. Summary

This is a motion proposal for the long-term R&D of the DFINITY Foundation, as part of the follow-up to this post: Motion Proposals on Long Term R&D Plans (Please read this post for context).

This project is about integration efforts of the IC with systems outside the IC: The Web, other blockchains, and end-user devices. The IC, like any other blockchain, cannot directly interact with machines on the Internet per se, e.g., by making API calls via HTTP(S). Integrations with outside systems are crucial for certain applications that rely on data from those systems and can help create value in multiple ways for the IC and smart contracts executed on it. The Main areas of integration are HTTP support for canisters, supporting other protocols for canisters to communicate with the “outside world”, oracle integrations, integrations with other blockchains, alternatives for cloud SDKs.

2. Discussion lead

Dieter Sommer

3. How this R&D proposal is different to previous types

Previous motion proposals have revolved around specific features and tended to have clear, finite goals that are delivered and completed. They tended to be measured in days, weeks, or months.

These motion proposals are different and are defining the long-term plan that the foundation will use, e.g., for hiring and organizational build-out. They have the following traits and patterns:

1. Their scope is years, not weeks or months as in previous NNS motions
2. They have a broad direction but are active areas of R&D so they do not have an obvious line of execution.
3. They involve deep research in cryptography, networking, distributed systems, language, virtual machines, operating systems.
4. They are meant to match the strengths of where the DFINITY foundation’s expertise is best suited.
5. Work on these proposals will not start immediately.
6. There will be many follow-up discussions and proposals on each topic when work is underway and smaller milestones and tasks get defined.

An example may be the R&D for “Scalability” where there will be a team investigating and improving the scalability of the IC at various stages. Different bottlenecks will surface and different goals will be met.

3. How this R&D proposal is similar to what we have seen

We want to double down on the behaviors we think have worked well. These include:

1. Publicly identifying owners of subject areas to engage and discuss their thinking with the community
2. Providing periodic updates to the community as things evolve, milestones reached, proposals are needed, etc…
3. Presenting more and more R&D thinking early and openly.

This has worked well for the last 6 months so we want to repeat this pattern.

4. Next Steps

Developer forum intro posted
1-pager from the discussion lead posted
NNS Motion proposal submitted

5. What we are asking the community

• Ask questions
• Read 1-pager
• Give feedback
• Vote on the motion proposal

Frankly, we do not expect many nitty-gritty details because these are meant to address projects that go on for long time horizons.

The DFINITY foundation’s only goal is to improve the adoption of the IC so we want to sanity-check the projects we see necessary for growing the IC by having you (the ICP community) tell us what you all think of these active R&D threads we have.

6. What this means for the existing Roadmap or Projects

In terms of the current roadmap and proposals executed, those are still being worked on and have priority.

An intellectually honest way to look at this long-term R&D project is to see them as the upstream or “primordial soup” from which more baked projects emerge from. With this lens, these proposals are akin to asking, “what kind of specialties or strengths do we want to make sure DFINITY foundation has built up?”

Most (if not all) projects that the DFINITY foundation has executed or is executing are borne from long-running R&D threads. Even when community feedback tells the foundation, “we need X” or “Y does not work”, it is typically the team with the most relevant R&D area that picks up the short-term feature or project.

[diegop]

Please note:

Some folks gave asked if they should vote to “reject” any of the Long Term R&D projects as a way to signal prioritization. The answer is simple: “No, please, ACCEPT”

These long-term R&D projects are the DFINITY’s foundation’s thesis at R&D threads it should have across years (3 years is the number we sometimes use internally). We are asking the community to ACCEPT (pending 1-pager and more community feedback of course). Prioritization can come at a separate step.

[jzxchiang]

This is probably a stretch, but if there’s some way to get something like WebSocket working with a canister, that would be huge…

[dieter.sommer]

1. Objective

The objective of this initiative is to integrate the Internet Computer with outside systems. We have identified multiple classes of external systems of interest in this domain: The Web 2.0, i.e., HTTP(S) and other standard protocols, other blockchains, and end user devices. As can be observed, these are quite different integrations collectively put together in this motion proposal. Ultimately, the goal of integrating the Internet Computer with such systems is to provide greater value to both the IC and the systems it integrates with.

2. Background

A blockchain is per se isolated from other systems outside of it. The standard defined interaction pattern with the outside world is that of users interacting with the blockchain by submitting transactions (or queries) and obtaining responses. The blockchain nodes cannot arbitrarily call out to servers on the Internet to retrieve data or send transactions to smart contracts on other blockchains. Replicas directly obtaining data from external data sources would likely break the consensus layer because different replicas would receive different responses (e.g., due to timestamps, unique ids, or changed data due to differences in the times when the requests were made). Blockchains solve the problem of providing external data to smart contracts through so-called oracles, the realization of which spans a spectrum from centralized to decentralized architectures. In every instance, though, external parties, the oracles, are required to provide data, thus complicating the overall trust assumptions of the smart contracts and increasing operational cost.
Making transactions to other blockchains is even more difficult in that transactions need to be signed and in the trust model of blockchains, network nodes cannot securely hold a signature private key, so threshold cryptography or trusted entities need to be used. Most integrations we see as of now are based on wrapping tokens of other chains through (centralized) bridges to enable DeFi use cases. General invocations between smart contracts on different chains are harder, but there are efforts of standardizing protocols for this purpose.
A quite different problem domain is the currently widespread use of proprietary centralized public cloud services through end users, e.g., through mobile apps that make use of these cloud services through easy-to-use SDKs provided by the cloud vendors. A good example for this are SDKs that offer cloud storage services to apps. There are no SDKs yet that would offer the functionality of the Internet Computer to mobile app developers in an easy-to-use manner. This should be addressed as part of this integration proposal. In contrast to integrating the IC with external systems, this is rather a use case where IC functionality is made available to end users via app SDKs, much like this is prominently done with cloud services these days.

3. Why this is important

The Internet Computer as a standalone system can already provide great value by offering a platform for running smart contracts at web-speed and even serving the frontend from the blockchain. However, many interesting types of smart contracts, such as decentralized finance (DeFi) smart contracts or decentralized insurance smart contracts, require external data in order to function. Such smart contracts are referred to as hybrid smart contracts as they, in addition to what is available on chain, also rely on external data for their logic. Thus, a solution that allows canisters to directly make HTTP requests to outside servers, an integration with existing oracle networks, or a proprietary oracle network is crucial for enabling vital use cases on the Internet Computer.
The integration with other blockchains can be considered the next level of integration with external systems and enables further use cases: If we would have broad inter-chain connectivity among the world’s top blockchains, smart contracts on all chains could interact with each other, thereby opening up a plethora of new use cases and leveraging network effects. An intermediate step towards this would be a broad availability of wrapped tokens of other blockchains on the Internet Computer. Besides the availability of external data, the availability of tokens of other chains is crucial for certain exciting high-value use cases such as decentralized exchanges (DEXs).

4. Topics under this project

We currently see the following topics of interest under the umbrella of this initiative:

• HTTP(S) support for canisters
  An important integration for the Internet Computer is the one with the current Web 2.0 infrastructure by allowing canisters to make HTTP(S) calls to servers on the Internet. This is a non-trivial problem in multiple ways:
  • The replicas of the IC each need to make the same call and obtain consensus over the result. However, results can differ, e.g., because of unique identifiers, timestamps, or slightly deviating numerical values, e.g., for quotes. Those differences need to be accounted for when bringing the HTTP call results through IC consensus.
  • Calls that change the state in the invoked external service, i.e., are not idempotent, pose another interesting challenge.
    Having an HTTP call mechanism enables a wide range of applications, e.g., oracle services directly integrated into the Internet Computer in a trustless manner, user notifications, and anything else that requires communication with servers on the Internet.
• Support of further protocols for canisters
  Besides HTTP, we intend to support other relevant protocols to integrate with Web 2.0 systems. To this end, we need to further investigate which protocols should be considered. Community input about which protocols to consider and the use cases this would solve will be greatly appreciated.
• Oracles
  An important part of this multi-year integration strategy are oracles. Oracles are crucial for providing data such as stock quotes, cryptocurrency, or any other relevant market data for DeFi, weather data for decentralized insurance applications, or any sort of interesting data feeds to smart contracts on the IC. The functionality of canisters making HTTP calls is closely related to oracles, however in a first iteration, HTTP calls will not provide all functionality of today’s most powerful oracle services. Thus, we want to also investigate an integration with existing oracle services, e.g., Chainlink, to leverage existing oracle ecosystems that are readily available and immediately benefit from the accompanying network effects.
• Blockchain integrations
  Particularly the domain of DeFi applications, e.g., decentralized exchanges (DEXs), integrations with other blockchains and specifically their respective native cryptocurrencies, besides Bitcoin and Ethereum, is of relevance. The Foundation does not have the resources to integrate with all other blockchains of interest, and therefore considers, as part of this motion proposal, to provide a blueprint or framework to enable the community to implement such integrations on their own or to use upcoming integration services. This will open up the Internet Computer for a plethora of exciting DeFi use cases that are not yet possible in their entirety.
• End user device integration: Cloud SDK alternatives
  Another quite different topic in the area of this motion proposal are blockchain alternatives to the cloud integration SDKs for mobile operating systems with API compatibility. Such cloud SDKs are currently available on the iOS and Android mobile platforms and allow users to, for example, store data on legacy cloud systems. Providing an API-compatible alternative to those SDKs would allow users to (almost transparently) use the same APIs (or a subset thereof) to perform the same functionality on the blockchain instead of the traditional cloud. A specific use case here would be storage of data. This could help users move from the Web 2.0 world of proprietary cloud stacks to an open blockchain world with many of their daily computing needs. This is quite different to the other integrations as it is rather a use case of the Internet Computer in the context of replacing traditional cloud computing technology.

5. Key milestones (and technical solution if known)

As this is a multi-year R&D initiative, we can currently not yet provide the concrete milestones for the individual features. Each feature will require milestones for the finalized system design, a PoC or MVP implementation, and further milestones for specific core functionalities. We plan to take an agile approach for the initiative in that we prioritize the functionalities, together with the community, and provide implementations early on to provide best possible value to the community. This will imply an agile approach of delivering MVPs initially that provide basic
functionality and later expanding the functionality based on what is most needed.

6. Discussion leads

@dieter.sommer, @THLO, @timo

7. Why the DFINITY Foundation should make this a long-running R&D project

The features addressed in this initiative motion proposal cover a broad spectrum of functionality and require substantial R&D efforts to bring them to the market. Basic functionality of the features may be relatively fast to implement, but more advanced functionality will require time for the underlying research efforts to develop feasible solutions to the problems. Based on those, system designs and implementations thereof need to be produced. Considering the expected breath and depth of this initiative, we think it can only be tackled through a multi-year initiative that is operated in an agile manner.

8. Skills and Expertise necessary to accomplish this (maybe teams?)

Due to the complexity of the initiative, we require teams with a broad selection of skills as outlined next:

• System design
• System level software engineering
• Algorithms, complexity
• Probability theory
• Cryptography
• Technologies behind different blockchains
• Deep understanding of Internet Computer consensus
• Oracle technologies
• Mobile cloud SDKs
• API design

At least the following teams are likely required:

• Research
• Networking
• Consensus
• Message Routing
• Execution
• NNS
• Security
• SDK

9. Open research questions

• Consensus protocol extension for agreeing on HTTP(S) responses
• Response aggregation in consensus to realize oracle functionality without oracles
• Architecture for integrating HTTP(S) calling with the IC protocol stack and particularly the consensus layer
• Threshold cryptography
• Algorithm research required for the integrations (e.g., management of connections to other blockchain nodes, handling of incoming blockchain blocks, block processing, protocol between IC components to request and receive blocks)
• Analogous questions for different protocols to integrate (e.g., streaming protocols)
• Evaluation of oracle networks and definition of integration architecture
• General approach / blueprint to integrating with other blockchains
• Security modeling and proofs

10. Examples how the community can contribute to the project

• Review proposals
• Provide feedback on the dev forum
• Submit NNS proposals
• Review technical designs
• Contribute to technical designs
• Review code
• Contribute code
• Participate in workshops on the initiative or its features
• Help us prioritize for the best benefit of the community

11. What we are asking the community

• Review comments, ask questions, give feedback
• Vote accept or reject on NNS Motion

[dieter.sommer]

I am Dieter Sommer, a member of the research team at DFINITY. I am responsible for this long-term R&D motion proposal and looking forward to the discussions with you on this exciting topic of integrations of the Internet Computer with the outside world.

[jzxchiang]

Thank you for the detailed proposal. It is a massive undertaking, and I think the proposal does a good job in summarizing ideas I’ve heard proposed on the forum over the last couple of months.

There are no SDKs yet that would offer the functionality of the Internet Computer to mobile app developers in an easy-to-use manner. This should be addressed as part of this integration proposal.

Can’t +1 this enough. Mobile and web SDKs like Firebase make development much more attractive, not to mention easier. The agent-js library is a great start, but may still be too low-level for some users. As you point out, storing files could be a compelling first use case of a Firebase-like IC client SDK.

Every project is reinventing the wheel on how to chunk and serve blobs from canisters. Ideally, they could just plug into a high-level SDK and just pass it a file path. This will be even more important once storage subnets are available IMO.

The replicas of the IC each need to make the same call and obtain consensus over the result.

I wonder how much of the BTC integration code can be reused for this. How does that code deal with consensus from calls made to the BTC network?

One simpler use case could be push notifications. In this case, the response to an external call is less important than the request. Fire-and-forget use cases like this may be easier to implement as a POC? Not sure. Right now, IC dapps either rely on centralized services for push notifications, or perform expensive polling on the canister. Neither are ideal.

[jzxchiang]

Blockchain integrations
Particularly the domain of DeFi applications, e.g., decentralized exchanges (DEXs), integrations with other blockchains and specifically their respective native cryptocurrencies, besides Bitcoin and Ethereum, is of relevance.

I’d also be curious your thoughts of Polkadot and its parachains, and whether you think what value a potential Polkadot integration (if any) would add.

For example, it seems like the IC is planning on directly integrating with BTC and ETH without need for an intermediary like Polkadot. I wonder why that’s technically feasible with the IC and not with other L1s (if that really is the case).

[dieter.sommer]

@jzxchiang

Every project is reinventing the wheel on how to chunk and serve blobs from canisters. Ideally, they could just plug into a high-level SDK and just pass it a file path.

Exactly, that’s how it should be done ideally to offload work from our developers to SDKs.

@jzxchiang

I wonder how much of the BTC integration code can be reused for this. How does that code deal with consensus from calls made to the BTC network?

Parts of the work on the networking layer can be reused, one component even completely. On the consensus side, the HTTP(S) feature requires quite a different integration for the reason of not being self-validating payloads as in the case of Bitcoin. I.e., consensus needs to agree on the response which requires gossiping and signing responses and providing a response to consensus only once it has reached a threshold of supporting replicas. This is more complex than the ingress-like handling we do for Bitcoin blocks.

@jzxchiang

One simpler use case could be push notifications. In this case, the response to an external call is less important than the request. Fire-and-forget use cases like this may be easier to implement as a POC? Not sure. Right now, IC dapps either rely on centralized services for push notifications, or perform expensive polling on the canister. Neither are ideal.

The simple notification scenario would be captured well as part of the extension to allow for a Customizable quorum (“unsafe mode”) with reduced security. See the HTTP(S) topic on the forum (Enable canisters to make HTTP(S) requests) for ongoing discussions on this.

If you want true support of an external service for making state-changing POST requests with the envisioned model, the external service will need to be able to handle n requests (1 from each replica) intending to achieve the same and only execute it once. Or you put a proxy doing this in front. There are multiple options with different tradeoffs here for connecting a blockchain with a traditional service.

[dieter.sommer]

@jzxchiang

I’d also be curious your thoughts of Polkadot and its parachains, and whether you think what value a potential Polkadot integration (if any) would add.

Interesting thought!

Polkadot is an integration layer between many blockchains. Thus, by connecting with Polkadot, we might gain a much broader integration basis with other chains that are already integrated with Polkadot in one go. This is definitely something that we should discuss further!

@jzxchiang

For example, it seems like the IC is planning on directly integrating with BTC and ETH without need for an intermediary like Polkadot. I wonder why that’s technically feasible with the IC and not with other L1s (if that really is the case).

Direct integration without intermediaries requires one crucial functionality: Threshold signing. Only with this functionality one blockchain can create transactions for another blockchain without a (trusted) intermediary. Anything else you need is a (maybe rather complex) implementation effort to get the communication between the two chains going accordingly.

I would not claim that no one else in the world can do this. A few others have attempted this already, but to the best of my knowledge there have been security issues with their threshold signing algorithm or implementation thereof. Threshold ECDSA is a hot topic currently with lots of efforts going on.

I would also claim that it is not feasible for the IC to build a native integration for every other blockchain around. In my view, we will have Bitcoin and Ethereum natively integrated and others via the more traditional approach of using bridges.

There is a nice effort ongoing by Psychedelic (formerly Fleek) of building an ETH ↔ IC bridge called Terabethia:

• Terabethia: Bridging Contracts & Assets Across Ethereum and the Internet Computer 🌉 | by Terabethia | Terabethia | Medium
• https://terabethia.ooo/

This will, in my view, be an important booster for DeFi on the IC as it is the first broad integration with Ethereum and allows for transferring tokens over to the IC. Thanks to the team at Psychedelic for this great work!

[diegop]

Proposal is live! https://dashboard.internetcomputer.org/proposal/35637

[dieter.sommer]

Happy New Year!

Let me point you to an interesting discussion related to integrating with Bitcoin-derived blockchains on another topic in this forum:

Integrate all Bitcoin FORKs on ICP [BCH, DOGE, DASH, etc...]

Hi, as I am not fluent in english, i am sorry if my english is not good, I just hope you’ll understand my idea. As Ethereum has already its own smart contract, and as it’s a lot of work to integrate it in ICP, I wonder why we do not integrate all Bitcoin FORK on ICP First… As these crypto (BTC Forks) are similar, maybe I am wrong, but I though it could mean less work to do to integrate them once it’s done for Bitcoin. Once Bitcoin is integrated to ICP, you could add BCH, DASH, DOGE, Maybe LTC,…

This is closely related to the topic of “Blockchain integrations” in this proposal and there’s a good discussion going on.

[diegop]

Relevant NNS Motion proposal: Enable canisters to make HTTP(S) requests - #37 by diegop

[Tyler]

That’s great! NNS

[diegop]

Proposal is live: Internet Computer Network Status

[JaMarco]

@dieter.sommer

A few others have attempted this already, but to the best of my knowledge there have been security issues with their threshold signing algorithm or implementation thereof. Threshold ECDSA is a hot topic currently with lots of efforts going on.

Do you have anymore insight on the other platforms that have attempted to implement threshold signatures and what issues they have had doing so? And also how Dfinity have resolved those issues in their implementation?

[dieter.sommer]

This is a quick answer from the top of my head: Various other projects have implemented threshold ECDSA as well, but to the best of my knowledge, all of them have some shortcomings in properties that make the resulting system problematic for real-world use. And I recall to have seen a couple of exploits, at least one at the protocol level where the private key could be reconstructed after a (rather small) number of computed signatures, of threshold ECDSA implementations as well.

One notable project that has implemented threshold ECDSA is ThorChain. From what I remember, their approach for threshold signing assumes that network communication is synchronous and that the protocol stops working when one node stops working, while our scheme relies on asynchronous networks and degrades gracefully (up until some point, of course) when nodes stop working.
(see also here: https://www.reddit.com/r/dfinity/comments/tr6wic/what_is_the_difference_between_canister_ecdsa_and/)

@victorshoup, the inventor of our threshold ECDSA protocol, can definitely give some further details on this question.

[hokosugi]

In 2020, ThorChain seems to have been upgraded from GG18 to GG20 , and GG20 seems to be capable of asynchronous communication, so there may be no more differences in asynchronous communication.ThorChain is specialized in DeFi, with ICPs that allow you to write smart contracts freely. While there are differences, I would like to know if there are significant differences from a technical point of view.

[diegop]

reasonable question. Let me share with folks know know more and see if they have any insight worth sharing.

[hokosugi]

Any replies?
Is it safe to assume that this is almost the same as the GG20 used by ThorChain, even though there are differences in methods? I consider the package with the limited full node of UTXO only as well as the threshold ECDSA to be a strong point, but I would like to know if there is something selling point with ECDSA alone.

[diegop]

I think my message got buried. I’ll post on the team slack again.