When I’m backfilling data into my production canisters using a nodejs agent-js client (with a promise concurrency of 10), I often run into this error:
FetchError: request to https://boundary.ic0.app/api/v2/canister/<canister_id>/read_state failed, reason: connect ETIMEDOUT 139.178.86.114:443
at ClientRequest.<anonymous> (/Users/.../node_modules/node-fetch/lib/index.js:1491:11)
at ClientRequest.emit (node:events:390:28)
at ClientRequest.emit (node:domain:475:12)
at TLSSocket.socketErrorListener (node:_http_client:447:9)
at TLSSocket.emit (node:events:390:28)
at TLSSocket.emit (node:domain:475:12)
at emitErrorNT (node:internal/streams/destroy:157:8)
at emitErrorCloseNT (node:internal/streams/destroy:122:3)
at processTicksAndRejections (node:internal/process/task_queues:83:21) {
type: 'system',
errno: 'ETIMEDOUT',
code: 'ETIMEDOUT'
}
I’m pretty sure this is due to the subnet-level rate limits that boundary nodes apply to all requests. These limits are pretty small.
How do most people handle this? Do I just retry this myself using something like node-fetch-retry
? The default retry logic in agent-js doesn’t handle socket errors like this unfortunately…