Intermittent Success and Failure in Sending get_transaction to Solana via Http Outcall

Developers
1

When sending a get_transaction request to Solana via Http outcall in a canister, it sometimes succeeds and sometimes fails. The error returned when it fails is similar to this: CanisterError = "query transaction rpc error: RpcRequestError(\"(SysTransient) \\\"No consensus could be reached. Replicas had different responses. Details: request_id: 4118936, timeout: 1724984550981015457, hashes: [901713c42ac950f63206b43633e8a7a267d0c778b7d7db9de3902288ca12ecf4: 16], [dd3dc11a9a468c2b41614963a67c84750a9ef0f94daad0782503d232166e7c8c: 10]\\\"\")"

I analyzed the response content of get_transaction. For a finalized signature, the response should be deterministic, and I also implemented a transform function to remove the HTTP headers. Normally, this should prevent consensus errors. Are there any other reasons that might cause consensus errors?

2

Based on the error you sent it seems that your canister is on pzp6e (16+10 responses = 26 nodes) and the request was done on the 30th August 02:22 UTC (from the timeout timestamp). Is this more or less correct?

From the metrics that the node provider share with us, I see that there are 429 (too many requests) errors at that time. If you are using a public RPC endpoint the requests limit are quite strict. So, either make fewer requests or use a paid RPC endpoint.

Let me know if my initial guess is wrong.

3

Yes,my canister is deployed in the Fiduciary subnet that has 28 nodes and I am using the Alchemy Free Edition rpc service,just for test.

4

From their rate limit docs it seems that the free limit is 330 compute units per second and a solana get_transaction API call takes 59 compute units. So if you send six or more requests at the same time it can happen that you hit the rate limit.

As mentioned try to use a paid service, do less requests or implement a backoff strategy in your canister to stay within the limit.

5

The same error occurs with getLatestBlockhash:

variant {
Err = record {
method = “get_latest_blockhash”;
reason = variant {
CanisterError = “request latest block hash error: RpcRequestError("(SysTransient) \"No consensus could be reached. Replicas had different responses. Details: request_id: 4173431, timeout: 1725431941258451555, hashes: [ba9ee3918be201e8cb62c34658b4dce1216cf7581d6150d4ab895077dab2b037: 14], [1a7ecc911e30cda01b6880368cdfbc7f64786798664c59cdb434f442dd435062: 14]\"")”
};
}
}

The getLatestBlockhash API requires only 7 CU, and if 28 nodes request only 196CU at the same time, it will not reach the 330CU limit.
By The way, when will The Reduced quorum Feature be available?

6

The No.2 response:(
variant {
Err = record {
method = “get_latest_blockhash”;
reason = variant {
CanisterError = “request latest block hash error: RpcRequestError("(SysTransient) \"No consensus could be reached. Replicas had different responses. Details: request_id: 4201318, timeout: 1725611448089484570, hashes: [3557145a55dc23cf28ca369bd86fc3c01fd1abaf24cd861443361f15bd6e4d07: 16], [d0d744e43fc1587e71c5c9d4041f402547ab4ca92305d2f1075802b5772166ec: 11]\"")”
};
}
},
)
The No.13 response:(
variant {
Err = record {
method = “get_latest_blockhash”;
reason = variant {
CanisterError = “request latest block hash error: RpcRequestError("(SysTransient) \"No consensus could be reached. Replicas had different responses. Details: request_id: 4201343, timeout: 1725611628447305702, hashes: [27e762d216849729a9e601708c9dbca5dc26f7721605585f0721a57e650172a0: 14], [0f5198736ab77e9d8d6f6a25568e41ebd52ce0f36c3f055a4c965b37ad79864a: 14]\"")”
};
}
},
)
The No.15 response:(
variant {
Err = record {
method = “get_latest_blockhash”;
reason = variant {
CanisterError = “request latest block hash error: RpcRequestError("(SysTransient) \"No consensus could be reached. Replicas had different responses. Details: request_id: 4201349, timeout: 1725611658296118966, hashes: [6a0ed596e82aea651e8898e56884d8d5e1991fcbd3d16260ce4c5d6e98eb63ea: 18], [f334dfbd1f1c9fcf2ee09e8e2fbdd8d79f778ba8666599d4c5d21264c5de5301: 10]\"")”
};
}
},
)
The No.20 response:(
variant {
Err = record {
method = “get_latest_blockhash”;
reason = variant {
CanisterError = “request latest block hash error: RpcRequestError("(SysTransient) \"No consensus could be reached. Replicas had different responses. Details: request_id: 4201377, timeout: 1725611734393712094, hashes: [8bb269221c67d1bfe6f6f1e040566435ef69a0326072c85f4c1917e57c6bacd8: 15], [fb59db837ec97476fe1eb195c8dc032ce769c63224a6dfa5815d2965bc97c3d3: 12]\"")”
};
}
},
)

The No.23 response:(
variant {
Err = record {
method = “get_latest_blockhash”;
reason = variant {
CanisterError = “request latest block hash error: RpcRequestError("(SysTransient) \"No consensus could be reached. Replicas had different responses. Details: request_id: 4201384, timeout: 1725611778458072012, hashes: [7e6bbab8309bb641a923f53a9c19aa2666235f23927b575f0cc476b055dc189f: 17], [d713fa558fe3b581bec717ef0704312bde9c4838670f41b2bc059083e4ad719f: 10]\"")”
};
}
},
)
The No.29 response:(
variant {
Err = record {
method = “get_latest_blockhash”;
reason = variant {
CanisterError = “request latest block hash error: RpcRequestError("(SysTransient) \"No consensus could be reached. Replicas had different responses. Details: request_id: 4201392, timeout: 1725611860043241235, hashes: [f58ef04160bacb2cd0c94eb85c390423d1ceee2780a9e081ca03eb7f3711f510: 15], [f888ba2052d94a7bb818e1ff27c07191b1352c3c018287e62ca83e7445697cc4: 13]\"")”
};
}
},
)
The No.30 response:(
variant {
Err = record {
method = “get_latest_blockhash”;
reason = variant {
CanisterError = “request latest block hash error: RpcRequestError("(SysTransient) \"No consensus could be reached. Replicas had different responses. Details: request_id: 4201397, timeout: 1725611891871040268, hashes: [554a150417b2df7cd1f05c54b08816ea34ab01e33c3b52833701f4955749b2d5: 14], [1d7a9babfaa2975c8564597b6978a981f2af7cf3bc161f9091025213852380e2: 14]\"")”
};
}
},
)
The No.37 response:(
variant {
Err = record {
method = “get_latest_blockhash”;
reason = variant {
CanisterError = “request latest block hash error: RpcRequestError("(SysTransient) \"No consensus could be reached. Replicas had different responses. Details: request_id: 4201412, timeout: 1725611998712047453, hashes: [cc5c29c76ca8256793cdba42f145b14cbe67e8943e7bfc3b079b1bc8097ccf1c: 16], [5595f3c7b41b65e29eb5d7179ae0e2972e9f5a8314457a257afb54c4f6a93718: 12]\"")”
};
}
},
)
The No.41 response:(
variant {
Err = record {
method = “get_latest_blockhash”;
reason = variant {
CanisterError = “request latest block hash error: RpcRequestError("(SysTransient) \"No consensus could be reached. Replicas had different responses. Details: request_id: 4201421, timeout: 1725612065774025714, hashes: [60251e2439088dc1ae1c53c817c47aab9ffb2c5bf9638bfd11916eca832eed44: 18], [5cd587a9d4b9b9c6a0d68c0c530d824e904a8964e82df55d655e279de7e955de: 10]\"")”
};
}
},
)
The No.45 response:(
variant {
Err = record {
method = “get_latest_blockhash”;
reason = variant {
CanisterError = “request latest block hash error: RpcRequestError("(SysTransient) \"No consensus could be reached. Replicas had different responses. Details: request_id: 4201442, timeout: 1725612127447419871, hashes: [b471c063235cbe4c9426b6045767eaf9db93835d64712fd9030999ca54b97dee: 17], [e42aaee2e13d5c05045043a0e22d4c4d50490f68f27e67331949b692bb1f2e12: 10]\"")”
};
}
},
)
The No.50 response:(
variant {
Err = record {
method = “get_latest_blockhash”;
reason = variant {
CanisterError = “request latest block hash error: RpcRequestError("(SysTransient) \"No consensus could be reached. Replicas had different responses. Details: request_id: 4201460, timeout: 1725612242591176972, hashes: [a1a1f4f1a2666110de7150584a207a6a747ba7c70a27d725bb920fdf45299062: 15], [f76d078c83b6306f5a842659754c983e586eefe4b8cae3d28ad4dd29870c0474: 10]\"")”
};
}
},
)
The No.84 response:(
variant {
Err = record {
method = “get_latest_blockhash”;
reason = variant {
CanisterError = “request latest block hash error: RpcRequestError("(SysTransient) \"No consensus could be reached. Replicas had different responses. Details: request_id: 4201582, timeout: 1725612789397868611, hashes: [a6dcc5820215aaf522322e8cbdf20899ba5d98012a8b3a70c1b9f13582542fb3: 15], [2f3ae4ab46d966175e0d9e97562335180aa58774711893a37a42dba7d8dc9c7f: 13]\"")”
};
}
},
)
The No.91 response:(
variant {
Err = record {
method = “get_latest_blockhash”;
reason = variant {
CanisterError = “request latest block hash error: RpcRequestError("(SysTransient) \"No consensus could be reached. Replicas had different responses. Details: request_id: 4201604, timeout: 1725612886614822090, hashes: [fb6770911d67b3c2a0460d3ded1967c517e2332184b4dce042ada579f6b94d75: 17], [9c3d2b91c233c76db5be7a177065db979cd14f46d5147e0d47f74405fd7e37d0: 11]\"")”
};
}
},
)
The No.96 response:
The No.97 response:(
variant {
Err = record {
method = “get_latest_blockhash”;
reason = variant {
CanisterError = “request latest block hash error: RpcRequestError("(SysTransient) \"No consensus could be reached. Replicas had different responses. Details: request_id: 4201656, timeout: 1725613016890405670, hashes: [f9c3cd9ef2c50f0035464a35098854a02a0bde257334ef0a54c0a1f05cfeff43: 14], [4f2342c444f1b85c03431a48ad6a13459d8270343689e677d77e9c62371499ac: 14]\"")”
};
}
},
)

@tim1
I use cli “dfx call canister $SOLANA_ROUTE get_latest_blockhash ‘()’ --ic” accesses solana rpc api, loops 100 times, “No consensus could be reached …” This error occurred 14 times. By checking alchemy log, the API getLatestBlockhash did not reach the request limit. Could you please help me find out the reason?

7

Sorry for replying late I missed the notification.

By The way, when will The Feature be available?

I don’t think it will be available soon. There is currently no active work being done on that feature. While it sounds simple it involves quite a lot of work and we would need to rearchitect many parts.

What can happen with API providers is that they have multiple solana nodes in the backend and due to timing issues it can happen that you receive many different responses. For example if I run this command:

for i in {1..25}; do (curl  https://solana-mainnet.g.alchemy.com/v2/MY_API_KEY -X POST -H "Content-Type: application/json" -d '{"id":1, "jsonrpc":"2.0", "method":"getLatestBlockhash", "params":[{"commitment":"processed"}]}' &); done; wait

I see many different block hashes. So make sure that the API you use is deterministic For example by providing a block number.

Also make sure you always strip the http headers in the transform function since they are a common source of non determinism.