Simplifying Integration of data platforms (such as CMC): Interface Converter Tool

[ICPSwap]

Hello! Greetings from the ICPSwap Team.

We are excited to share that we are contemplating the development of an “Interface Converter Tool” to enrich the IC ecosystem. Once this “Interface Converter Tool” is completed,
integrating with new data platforms will be as straightforward as configuring and deploying based on the platform’s interface documentation.

It all started last year through our collaboration and partnerships with esteemed data platforms! To integrate with such data platforms, we initially had to create an agent canister tailored to each platform’s unique interface. Recently, with our partnership with CoinMarketCap (CMC, https://coinmarketcap.com), we find ourselves in a similar situation, needing to re-develop another agent canister that aligns with CMC’s interface standards to integrate CMC’s API and connect it with the ICPSwap data canister as per the interface documents ([Public] Ideal API Endpoint - Google Docs) CMS has provided with us.

Why not once-and-for-all? Otherwise, we might have to repeat the same operation again and again in the future!

Therefore, this repetitive work has inspired us, the ICPSwap team, to consider a more universal solution: the “Interface Converter Tool” as mentioned above at the beginning of this post. This tool aims to simplify the integration process for all IC projects, eliminating the need for redundant, project-specific agent canisters.

We suspect that many within the IC community share our need for a streamlined method to connect centralized data platforms with project data canisters. Once our tool is complete, we are committed to open-sourcing its code for the benefit of the entire community.

To ensure that this tool serves the collective needs of the IC ecosystem, the ICPSwap team invite your input. Please offer us your invaluable thoughts! Do you have a similar need for such as tool? What is the format of the interfaces of the data platforms you want to integrate with? What is the DID for your project’s data storage? Your feedback will be instrumental in helping us create a more versatile and effective tool that will invigorate the IC ecosystem.

We eagerly await your thoughts and thank you in advance for your support and contributions!

Warm regards,
The ICPSwap Team

[gatsby_esp]

What is this for? Real time price of tokens on the IC ecosystem from coinmarketcap?

[ICPSwap]

Hello, sir. Display trading pairs, trading volume, prices,1h change, 24h change, and other information for tokens within the IC ecosystem on Coinmarketcap.

[slq016]

good,
come on .
icpswap used very easily .
i feel icpswap login need add me Wallet .

[ICPSwap]

Provide more detailed updates:

I — Problem

Recently, we have been docking with some centralized data platforms. We find that centralized data platforms always use HTTP interfaces to collect data, but ICPSwap is developed based on IC and does not provide an HTTP interface, and the data model is also different from that of the data platforms. So we need to develop a proxy canister to convert the HTTP interface into an WASM interface and perform data conversion. And similar work has been done when we docked with another data platform last year. Due to the different interfaces and data models between the two data platforms, these two proxy canisters cannot be reused. And if we need to dock with other data platforms in the future, we still have to do the same work again.

So we believe that there are also many projects in IC ecosystem like us who may have the need to connect to centralized data platforms. Therefore, the deployment diagram will look like the two pictures below, which will result in a lot of repetitive development work for everyone.

picture A

picture B

Look at the picture A, it’s a real diagram deployed in IC, the data canister belongs to ICPSwap, and three data platforms get data from data canister through three API canisters.

And if there are two data canisters should be connected like picture B by two data platforms, four API canisters should be deployed. But the core logics in these canisters are same, and the difference between the four API canisters are the interfaces and data models. So if we can keep the core logics and make the interface and data model configurable, we can save more time and code.

II — Solution

We plan to develop a centralized data transformation (or Interface Converter Tool) platform based on IC.

The platform can dynamically configure the HTTP interface and read data from any other projects in IC with 0 code. So that it can strengthen the interaction ability between the IC ecosystem and other ecosystems, and enhancing the vitality and influence of the IC ecosystem.

Because the project will be open source, other projects can also use the code to independently deploy their own platform and configure their data conversion services.

III — Architecture

i — Configuration

Configuration Service includes Upstream Management, Model Management, and API Management.

• Upstream means canisters here. Upstream Management maintains principal and methods of the data providing canisters.

• API Management maintains HTTP interfaces which provide external services.

• Model Management includes Argument Model, Data Model and Upstream Model.

  • Argument Model maintains arguments in API interface.

  • Upstream Model maintains input value models of canister methods.

  • Data Model maintains the data structure which get from canisters and will be returned to the API interface.

• Field Mapping

  • Argument Model need to be converted into Upstream Model by field mapping, to make sure that correct input values can be passed into the Upstream Management.

  • Output data from Upstream Management should be converted into Data Model by field mapping, to make sure that correct output values can be returned to API Management.
    

    

ii — Endpoint

When an HTTP request enters the endpoint canister:

Firstly, it goes through the auth module to check the access privilege and save the audit log.

Then parses the request path to the upstream interface, and converts the request parameters into input parameters of the upstream interface, and make an WASM call.

After obtaining the return value of the upstream interface, the return value will be parsed into a particular data model by configuration, and return to the caller.

IV — Load Balance

Endpoint canisters can also be horizontally expanded by replicating multiple canisters to provide services. Different service strategies can be selected based on different type of clients.

i — Load Balancing Software

Data platform can use centralized load balancing software like Nginx, to perform server-side load balance on endpoint canisters.

i i — Registry Center

When user creates an endpoint canister, the canister will be registered in registry center. And data platform can get endpoint canister list from registry center, and implement their own load balancing logics.

We eagerly await your thoughts and thank you in advance for your support and contributions!

[ICPSwap]

@slq016

good,
come on .
icpswap used very easily .
i feel icpswap login need add me Wallet .

Thank you so much for your support!

The Me Wallet is now available on ICPSwap! You can use the Me Wallet to connect to ICPSwap.

[gatsby_esp]

what timeline do you expext to have this developed and connected to coinmarketcap? thanks

[ICPSwap]

If we plan to develop this, the estimated development timeline would be 3-4 months.

Meanwhile, we are in communication with CMC and working on integration. We will aim to list ICPSwap’s data on CMC ASAP.

[cryptoschindler]

@cakemaker1 @infu @simpson

do you have any feedback on this approach? is this something that would be useful for you as well?

[gatsby_esp]

Is ICPSwap already on CMC?

[infu]

This will definitely be useful for services outside of the IC and ones that don’t require maximum security and speed. I still want DEXes to provide data from the source canister with ICRC standards.

[ICPSwap]

ICPSwap is already listed on it, but the data integration is still in progress.

[BABYAROF]

We are very excited by your news, we have always had challenges to list the BABY AROF token data on public platforms, also the public platforms that provide the data often did not support this blockchain due to the different ICP structure. .

This is a big step, I firmly believe that your action is a big and beneficial step for the entire ICP community. This increases engagement and drives more developers to ICP.

Best wishes to your team, we have always watched your team’s efforts closely. Thankful

[ICPSwap]

Thank you so much for your kind words and support!

[srcChain]

@ICPSwap, I know I am late to the party, but what is the update on this? I think it is a great idea and would benefit other projects in the community tremendously.

[ICPSwap]

Hello, sir, you’re not late at all. Thanks for checking in. Please take a look at this open-source code: GitHub - ICPSwap-Labs/ic-data-platform

Glad to hear your approval.

[skilesare]

This is a super cool tool. I think I understand what is happening here, but could you give a simple, real-world example of how this is used? Is it kind of like a stand in for something like rosetta server, but for a broader set of data?