HTTPS outcalls no response

mozhiyun · July 4, 2024, 6:14am

Hello, everyone，when referencing the implementation of cketh to perform HTTPS outcalls to access two nodes, I encountered some issues，how should this be handled?

Sometimes, accessing Node 1 works fine, but accessing Node 2 gets no response. All errors have been matched and logged.
Often, data from Node 1 returns correctly, but data from Node 2 fails to reach consensus
Here is the translated error message:

“Failed to get logs from block 6_241_418 to block 6_242_217: ConsistentHttpOutcallError(IcError { code: SysTransient, message: "No consensus could be reached. Replicas had different responses. Details: request_id: 314732, timeout: 1720072156081316980, hashes: [0e3f620af75967218e9fde7d2080d7c8c8d6677ae505dd535a760ee892d048a0: 5], [5069345b26a166a00ce8d1a5563df3ef207bf8a33a71e7c0afdcbeb59c6b79fb: 4], [c2d568e60e242094a09433886990798a4c9d3407d41fcbcfd5be698a32c25152: 3]" })”
Here is the correct translation of the information：

“Got response of retries 1 (with 2053 bytes): {"id":10,"jsonrpc":"2.0","result":[{"address":"0xb21867b86787a5da03293ff2595b91a2b95f1f26","topics":["0x2e3668ee6cb513b98452cf0ef87b82e440b57c152dee3fe2339ff218ca7fb6a7","0x0000000000000000000000008d86f24500f2b639393a69947cb4d9a334ca395c","0x000000000000000000000000296065b60d30846061fd31080467edd4e1c77fcc","0x1dcb3ce99798d7c817eedf4b00a0af5bdb536505c5ffb6f47c3797ddc3020000"],"data":"0x0000000000000000000000000000000000000000000000000000000002625a00","blockNumber":"0x5f3ca3","transactionHash":"0x7a19d01256628b0873d280fa8f924ad96225006e03428ce4e3e792a4e30b214a","transactionIndex":"0x4a","blockHash":"0x43588b77692fee5bff5c1dec53b3cf71fe9dd5c33c148a0b9febbc7bac5ed481","logIndex":"0x93","removed":false},{"address":"0xb21867b86787a5da03293ff2595b91a2b95f1f26","topics":["0x2e3668ee6cb513b98452cf0ef87b82e440b57c152dee3fe2339ff218ca7fb6a7","0x0000000000000000000000008d86f24500f2b639393a69947cb4d9a334ca395c","0x000000000000000000000000705784d8a49deeb8d52dc745c84a2e73c3c34dc5","0x37d5282bd8ff94fe79d3d283aed69882e088a00d000000000000000000000000"],"data":"0x000000000000000000000000000000000000000000000000000000000000000b","blockNumber":"0x5f3ccc","transactionHash":"0x8c0326a2a9300e596bd91af54fa2f3e0b4a0a536d931cea32a8ecd959d0077e6","transactionIndex":"0x33","blockHash":"0x9bdb33b897952c986bea3376bdd7248ee44b1ae1316476022b278fa80a193ec2","logIndex":"0x64","removed":false},{"address":"0xb21867b86787a5da03293ff2595b91a2b95f1f26","topics":["0x2e3668ee6cb513b98452cf0ef87b82e440b57c152dee3fe2339ff218ca7fb6a7","0x0000000000000000000000008d86f24500f2b639393a69947cb4d9a334ca395c","0x000000000000000000000000705784d8a49deeb8d52dc745c84a2e73c3c34dc5","0x37d5282bd8ff94fe79d3d283aed69882e088a00d000000000000000000000000"],"data":"0x000000000000000000000000000000000000000000000000000000000000000b","blockNumber":"0x5f3cb4","transactionHash":"0x148fb6344244553202b757b1c558a523de0232e5c9bd0f7b76408e0a88c4b2d5","transactionIndex":"0xd","blockHash":"0xe23ede2ea13d56c4052de6244b021f6fe7e0823759b66a90289396d21546eac8","logIndex":"0xc","removed":false}]} from url: https://rpc.ankr.com/eth_sepolia with status: 200”,
@domwoe

zensh · July 4, 2024, 2:42pm

Are you missing idempotency handling?
Please refer to this: How to use HTTPS outcalls: POST | Internet Computer

We have implemented an idempotent reverse proxy service for this scenario. Feel free to use it!

mozhiyun · July 4, 2024, 3:23pm

Thank you. Question 2 may be the situation you mentioned. The ID increment by 1 failed when retrying (the specific reason why it failed cannot be understood. Please refer to the official implementation of cketh), which may lead to a lack of consensus. However, the situation of Question 1 is as shown below. Each request, including retries, will be incremented, but there is no record for the request with id=1293. This is something I can’t understand.

zensh · July 5, 2024, 3:13am

I checked the source code and it seems that idempotency handling is not implemented. Although most Ethereum JSON-RPC APIs are idempotent, because JSON object keys are unordered, multiple identical requests might not have equal serialized bodies even if they return the same JSON result.

github.com

internet-computer-protocol/evm-rpc-canister/blob/main/src/candid_rpc.rs

use std::str::FromStr;

use async_trait::async_trait;
use cketh_common::{
    eth_rpc::{
        into_nat, Block, FeeHistory, GetLogsParam, Hash, LogEntry, ProviderError, RpcError,
        SendRawTransactionResult, ValidationError,
    },
    eth_rpc_client::{
        providers::{RpcApi, RpcService},
        requests::GetTransactionCountParams,
        EthRpcClient as CkEthRpcClient, MultiCallError, RpcConfig, RpcTransport,
    },
    lifecycle::EthereumNetwork,
};
use ethers_core::{types::Transaction, utils::rlp};
use ic_cdk::api::management_canister::http_request::{CanisterHttpRequestArgument, HttpResponse};

use crate::{
    accounting::get_rpc_cost,

This file has been truncated. show original

The parallel_call in the ckETH minter requests multiple providers simultaneously but only returns the result from the first provider. This is likely the cause of the no response issue you mentioned.

github.com

dfinity/ic/blob/master/rs/ethereum/cketh/minter/src/eth_rpc_client/mod.rs#L156


      
                  futures::future::join_all(fut).await
              };
              MultiCallResults::from_non_empty_iter(providers.iter().cloned().zip(results.into_iter()))
          }
          
          pub async fn eth_get_logs(
              &self,
              params: GetLogsParam,
          ) -> Result<Vec<LogEntry>, MultiCallError<Vec<LogEntry>>> {
              // We expect most of the calls to contain zero events.
              let results: MultiCallResults<Vec<LogEntry>> = self
                  .parallel_call("eth_getLogs", vec![params], ResponseSizeEstimate::new(100))
                  .await;
              results.reduce_with_equality()
          }
          
          pub async fn eth_get_block_by_number(
              &self,
              block: BlockSpec,
          ) -> Result<Block, MultiCallError<Block>> {
              use crate::eth_rpc::GetBlockByNumberParams;

I believe the evm-rpc-canister should integrate with the Idempotent Proxy to avoid potential No consensus issues.

mozhiyun · July 5, 2024, 5:10am

Thank you very much. If you continue to track the parallel_call in the ckETH minter. In the end, it waits for multiple providers to respond and then compare the results. It does not only take the result of the first provider.

Problem 1 is that it cannot get the multiple responses, which makes subsequent operations impossible. Comparing the results, subsequent logs cannot be printed, so I don’t think this is the problem.

If problem 2 occurs, the result.len() normally print 2, and provider 1 can print the response normally, provider 2 can print error log that indicates it fails to reach consensus. But when problem 1 occurs, the log can only print provider 1’s response, subsequent logs cannot be printed and is lost.

mozhiyun · July 5, 2024, 9:01am

I switched to a single provider, but it’s still not working, even with provider that previously returned data normally.
Additionally, APIs like eth_getBlockByNumber and eth_feeHistory are working fine; it’s only eth_getLogs that’s experiencing the issue mentioned above！
This issue only started occurring yesterday. The same code did not have this problem before.
Could you please take a look at it?
@domwoe

Topic		Replies	Views
Evm rpc httpOutcallError No consensus could be reached Rust	13	137	August 25, 2024
HTTP POST outcalls consensus not working , only one is working Rust	12	573	August 4, 2023
HTTP outcall error SysTransient Connection refused Developers	1	188	February 28, 2024
Http Outcall local development http Developers rust	3	383	August 9, 2023
QUESTION: Is it okay to set `max_response_bytes` on a fire-and-forget HTTPS outcall to '0'? Rust	7	343	August 8, 2023

HTTPS outcalls no response

Related Topics