How to handle ETIMEDOUT errors?

When I’m backfilling data into my production canisters using a nodejs agent-js client (with a promise concurrency of 10), I often run into this error:

FetchError: request to https://boundary.ic0.app/api/v2/canister/<canister_id>/read_state failed, reason: connect ETIMEDOUT 139.178.86.114:443
    at ClientRequest.<anonymous> (/Users/.../node_modules/node-fetch/lib/index.js:1491:11)
    at ClientRequest.emit (node:events:390:28)
    at ClientRequest.emit (node:domain:475:12)
    at TLSSocket.socketErrorListener (node:_http_client:447:9)
    at TLSSocket.emit (node:events:390:28)
    at TLSSocket.emit (node:domain:475:12)
    at emitErrorNT (node:internal/streams/destroy:157:8)
    at emitErrorCloseNT (node:internal/streams/destroy:122:3)
    at processTicksAndRejections (node:internal/process/task_queues:83:21) {
  type: 'system',
  errno: 'ETIMEDOUT',
  code: 'ETIMEDOUT'
}

I’m pretty sure this is due to the subnet-level rate limits that boundary nodes apply to all requests. These limits are pretty small.

How do most people handle this? Do I just retry this myself using something like node-fetch-retry? The default retry logic in agent-js doesn’t handle socket errors like this unfortunately…

1 Like