ICP Mainnet HTTP Consensus Error: "No consensus could be reached. Replicas had different responses"

Developers
1

Issue Description

Working on Internet Computer Protocol (ICP) with HTTP outcalls to Anthropic API. The code works perfectly in local development environment but fails consistently on mainnet with consensus errors. Multiple replicas are returning different response hashes, preventing consensus.

Error Message

HTTP request failed after 3 attempts: (SysTransient, "No consensus could be reached. Replicas had different responses. Details: request_id: 636027, timeout: 1757578358780186775, hashes: [f2ca70c5509a0a70ee73ad6008c23c3ccf5e4f8d09fa5d26fc82431592a0558f: 1], [e6314eb84de09a895642b41178f12e63e5a55bbbb5f64401c8db7bbc2820a15c: 1], ...")

Current Implementation

Main Function:

#[update]
pub async fn ask_ai(prompt: String, system_prompt: String, quick: bool, api_key: String) -> Result<AiResponse, String> {
    let request_body = json!({
        "model": "claude-3-haiku-20240307",
        "max_tokens": 4096,
        "system": system_prompt,
        "messages": [{"role": "user", "content": prompt}]
    });

    let request = CanisterHttpRequestArgument {
        url: "https://api.anthropic.com/v1/messages".to_string(),
        method: HttpMethod::POST,
        body: Some(serde_json::to_vec(&request_body)?),
        max_response_bytes: Some(20_000),
        transform: Some(TransformContext::from_name("transform".to_string(), vec![])),
        headers: vec![
            HttpHeader { name: "Content-Type".to_string(), value: "application/json".to_string() },
            HttpHeader { name: "x-api-key".to_string(), value: api_key },
            HttpHeader { name: "anthropic-version".to_string(), value: "2023-06-01".to_string() },
        ],
    };

    let cycles = 230_949_972_000u128;
    let (response,) = http_request(request, cycles).await?;
    // Process response...
}

Transform Function:

#[query]
fn transform(raw: TransformArgs) -> HttpResponse {
    let sanitized_body = if raw.response.status == 200u16 {
        match String::from_utf8(raw.response.body.clone()) {
            Ok(body_str) => {
                match serde_json::from_str::<serde_json::Value>(&body_str) {
                    Ok(mut json) => {
                        if let Some(obj) = json.as_object_mut() {
                            obj.remove("id");
                            obj.remove("created_at");
                            obj.remove("model");
                            obj.remove("type");
                            obj.remove("usage");
                        }
                        serde_json::to_vec(&json).unwrap_or(raw.response.body)
                    }
                    Err(_) => {
                        let simple_response = json!({"content": [{"text": "Response parsing error"}]});
                        serde_json::to_vec(&simple_response).unwrap_or(raw.response.body)
                    }
                }
            }
            Err(_) => raw.response.body,
        }
    } else {
        raw.response.body
    };

    HttpResponse {
        status: raw.response.status,
        body: sanitized_body,
        headers: vec![],
    }
}

Environment

  • Local: Works perfectly

  • Mainnet: Consensus failures with different response hashes across replicas

  • API: Anthropic Claude API v1/messages endpoint

The transform function attempts to remove non-deterministic fields but replicas still generate different response hashes, suggesting the sanitization may be insufficient or there are other sources of non-determinism.

2

I’ll just answer briefly, as there are already multiple similar threads on the forum.

Most likely, your implementation works locally because you’re local replica/pocket-ic is performing a single call, while on mainnet the IC performs multiple calls to your HTTPS endpoint and tries to reconcile the answers to ensure they are valid.

In other words, one HTTPS call is executed multiple times, and every response must be identical.

This is generally resolved by using an idempotency key or a proxy (custom solution or using existing tools such as those of this answer).

Also note that your target should support IPv6.

In the future, there might be an option to target IPv4 and skip reconciliation. There should be a post about this on the forum as well.

3

So i need background tasks. Right?

4

I don’t know what a “background task” is. The response to a particular request provided by your endpoint must be replicable. Given the nature of the Anthropic API, I assume it is not replicable, which may explain the need for a proxy. However, if it is replicable, then ensure that an idempotency key and IPv6 are used.

For future improvements that may not require replicated calls and that also support IPv4, refer to this thread: Announcing Two Major Upgrades for HTTPS Outcalls: IPv4 Support + Non-Replicated Calls are now LIVE!

1 Like