Enable canisters to make HTTP(S) requests

There are no plans to enable this because it is already possible now. You can simple send the API key as a header or part of the query string, where the further is the recommended solution and the latter frowned upon in the Web security community.

The only caveat is that the API key (a secret) is stored within the replicas. This risk needs to be taken and is manageable: in case of expose of an API key, the adversary can deplete your query quota. The situation is worse in case you use your API key to authorize for writing data to a server. This caveat cannot be avoided, though, in the current architecture.

I share this worry after our investigations on rate limiting. But with the abovementioned way of using API keys you should be fine and have your own quota through the API key.

2 Likes

@h4rdcoder: Could you resolve your issue meanwhile, it seems to work for at least @lastmjs in the local environment. Can you share more details on the SDK version you are using, the call you are making and the error you receive? Maybe someone from the community our our engineering team can help you resolve the issue.

1 Like

Update

We need to currently address some inconsistencies between the interface specification and the implementation of the feature. Those have emerged during the implementation and need to be addressed now.

Regarding how to model the transform function in the interface spec, a larger discussion has come up. The current way of modelling it as

    transform : opt variant {
      function: func (http_response) -> (http_response) query
    };

is seen to be inconsistent with the way we usually handle canister methods that the replica calls, namely system calls.

The discussion is ongoing, see the above PR for the interface spec, and depending on the outcome, we may need to update the interface spec and implementation to reflect the decisions, leading to some delay. Those inconsistencies are also the reason we could not yet make a mainnet deployment of the feature as originally envisioned.

The main driver for a potential change is consistency within the interface spec which is considered highly important at DFINITY. Inconsistency is like a broken window in software engineering, for those who are familiar with this concept, as it invites for further inconsistency in the future, so should be avoided whereever possible. So we need to see where the discussion goes and whether this is perceived a major inconsistency and, in case, act accordingly and get it fixed.

4 Likes

To a naive outsider, the higher level issue in the conversation mentioned in the PR, looks like using the particular construct (function reference) to drive certain other changes ( more library support for rust, a step towards function closures etc).

While the change has to begin somewhere, not sure whether this (http_request_method) is the appropriate place to have this battle(of ideas).

1 Like

The original post mentioned requests made by replicas.

What are the benefits of using boundary nodes here? Regarding rate limits of a service it sounds better to block a subnet (replicas) than the entire IC (random BN) or region (closest BN).

The benefits of using boundary nodes would be that only boundary nodes have IPv4 addresses. We do not have sufficiently many IPv4 addresses to equip all IC replicas with IPv4 addresses. Thus, using boundary nodes is one avenue towards enabling IPv4 connectivity for the HTTP feature.

As you say, in terms of rate limiting, going via boundary nodes means that if a boundary node is blocked, it is not available for any replica to use as a proxy. But, as the number of boundary nodes would grow with the number of replicas, the problem would not be as pronounced as it seems.

1 Like

Knowing the involved people, I don’t think this is the case, I think think this is really about getting the best possible solution in place for the IC. Driving certain other changes would be more a side effect of this.

Regarding the discussions on the interface specification we had further discussions. Many thanks here to @nomeata for giving a valuable perspective, @ielashi helping us to come to a decision, and finally @bjoern for making a decision on the information on the table!

See here for the beginning of the discussion: Update ic.did by lwshang · Pull Request #75 · dfinity/interface-spec · GitHub

Great discussion with the outcome that we leave the specification as is, modelling the transformation function as query calls and keeping the system API minimal. The Rust implementation is updated to finally match the specification.

The final MR went into master yesterday and should make it into the Release Candidate for the upcoming release, starting rollout in the coming week. We will first enable the feature on a non-whitelisted subnet for further quality assurance, followed by a rollout throughout the IC.

13 Likes

Any updates on the rollout? Great to see it’s all coming together!

1 Like

The feature has been enabled on a first subnet of IC mainnet! :rocket:

The HTTP feature has been rolled out to and enabled on a first subnet (fuqsr) that is now used for testing the feature on IC mainnet. An HTTPS request to the URL https://example.com/ has been the first direct request ever being successfully executed from a smart contract to a Web 2.0 HTTPS server. :tada:

This was an important step to be executed before a full rollout of the feature on the IC. We will now perform further testing and, if everything works as expected, perform a larger rollout in the coming weeks. Note that the subnet the feature has been deployed on is not available to the public.

41 Likes

I’m confused, does the spec now require transform or transform_method_name?

The current Rust example uses transform_method_name but the current interface spec uses transform

1 Like

The spec is right, the code change to the sample dApp seems to not yet be in the repo.

3 Likes

Correct, spec-aligning pull request to the sample dapp is going through review.

3 Likes

What version of dfx will have the correct version being deployed then? dfx 0.11.0 is what we implemented in Azle and all of our tests pass. So I assume dfx 0.12.0 has the fully correct http_request interface?

@dieter.sommer do you know starting from which commit of the replica the correct version is included?

It is the most recent one, 0.12.0-beta.3, released earlier this week.

4bd164bbc600421a2e6f93feb41f3d1479bb8d0f

Stay tuned for more information tomorrow morning Swiss time!

1 Like

It’s not morning yet, but I can say: Latest release (EDIT: that’s SDK release) does NOT yet contain this commit. The next time we update the replica version the change will be included

@lastmjs, @Severin:
The 0.12.0-beta.3 contains an earlier replica version (9173c5f1b28e140931060b90e9de65b923ee57e6) than the one being rolled out in mainnet (4bd164bbc600421a2e6f93feb41f3d1479bb8d0f), but it has everything we need regarding the latest changes for the feature.
You need to install the beta.3 version, e.g., by using the following command:
DFX_VERSION=0.12.0-beta.3 sh -ci "$(curl -fsSL https://internetcomputer.org/install.sh)"

2 Likes