Make external (non-IC) HTTP request from backend canister?

[shan]

Hello. I’m looking to build an application that I would need to poll various external servers from a backend canister at fixed time intervals to fetch user data, and then store the result of that get request in my canister’s database, after processing it and doing a few other things.

Is this possible to achieve right now with the IC? If not, does anyone know if there are any plans to?

Thanks!

… and also

Unrelated question: does anyone have a link to a working Rust build setup example? I am running into so many OpenSSL compilation errors where it can’t find any c_long c_void etc. types from the openssl-sys crate for some reason. Think it’s related to openssl wasm-target support? This is specifically when I try to dfx deploy, I can build with cargo fine.

[cryptoschindler]

hey @shan ! welcome to the forum and great first question. maybe this helps for the first part? the whole thread might be of interest for you.

Can we somehow achieve or build a canister with a interface bahaving like a classic web server

The ic-http-lambda is just a side proof of concept of mine, to show what’s possible with an HTTP interface. The fact that I run it on AWS is immaterial, once a feature like that becomes official nobody will remember my little experiment Can I write something to fetch data from outside http APIs and save it to IC, or forward to telegram chat? No, there is no good story for doing HTTP requests from a canister; it is somewhat at odds with replicated deterministic execution. So no…

for the second part maybe @lastmjs can help?

[cryptoschindler]

he’s building this with rust

sudograph/sudograph

[shan]

Ooh, thank you! I’ll take a look

[shan]

Wondering if anyone has any information about this as of yet

Without something like WASI, making http requests to an endpoint not hosted on the IC cannot be supported out of the box.

Right now, the only solution I can see for solving this issue is to do a workaround where:

1. the user initiates a request to a owner-maintained centralized server () hosted outside of the IC
2. this server processes the request to fetch data from an external data source
3. the server sends the response from the external data source to the backend canister using an IC agent
4. the backend canister receives the request and emits an update saying that it’s done processing the request
5. the user is notified of their request success/failures via some kind of polling response from the backend canister

Or, for a real crap solution: to just make the request to the non-ic-hosted service directly from the user’s local machine via frontend asset deployment, and update the backend canister directly from the response. This is, as you might suspect, inviting so many security issues there is really no point in developing it as the trustworthiness of the data is practically zero.

TL;DR: It possible right now to have an external server which pushes data to the IC from whatever source, but to my knowledge, it is not possible to request data from an external server from within a canister itself.

I am wondering if Dfinity has announced any plans for how to request data from external resources from within a canister? Something similar to ChainLink’s oracle node system perhaps?

[skilesare]

You have to keep in mind that calling an external resource breaks the determinism of the IC. If 4 nodes called a URL and they all got different results for something as simple as a timestamp in the html/json, they would never reach a consensus!

The solution is something like you suggested except that the IC can’t even send the request. I can just create an internal flag and the external server needs to poll that flag and then go get the data when it sees the flag raised.

If you want to decentralize this/increase security then you need to set something like chainlink up where a set of validators agree on an external value.