Failed to fetch response: TypeError: Failed to fetch

Developers
22

I’d like to bring up a slightly different error that I’ve seen:
TypeError: fetch failed

This is not the same as the TypeError: failed to fetch issue mentioned in this post


This issue pops up almost daily, for the past few months (sometimes several times a day).

I have a job (not in browser) that polls a polls a canister on the IC once a minute. If I can’t reach the canister, I email myself the error.

Our infrastructure is set up such that it doesn’t really if communication with the IC fails intermittently (we just keep values in a queue and try a minute later), but the frequency of these errors is the concerning part - it is something I’ve been meaning to look into.

Here’s what my emails received looks like.

IMG_28D53035B983-1
IMG_28D53035B983-11170×2399 136 KB

I have no idea if this is on the agent or server side, but I’m curious if anyone else has run into this error.

1 Like
23

that’s not a custom error anywhere in the agent-js codebase. I think it’s from getting no response back from the boundary node

1 Like
24

@NathanosDev (or boundary node team) I wanted to follow up here to see if there’s any insights on your side that might help. I received 3 of these errors on September 4th (California, US time) at 8:32PM, 10:30PM, and 11:17PM. Let me know if that helps trace down the issue.

1 Like
25

Thanks! I’ll share this with the boundary node team.

26

We still receive these errors a handful of times a day, but their frequency seems to be increasing to a point that is borderline worrisome.

On January 8th between 14:33 and 14:56 UTC we polled the IC and received 16 errors out of that 23 minute span (only 7 calls were successful).

On January 9th between 10:43 and 11:13 UTC we polled the IC and received 21 errors out of that 30 minute span (only 9 calls were successful).

I’m assuming the errors received are related to this issue

Note that as of this post, we haven’t seen such an error in the past 16 hours.

27

@rbirkner

The error count is consistently down now (back to the rate we were seeing previously), but I still am receiving these errors 2-3 times a day. The last errors (2) were received on January 11th at 13:27 and 13:28 UTC.

28

Hey @icme

It would be great if you could share more details with us that help pinpoint the problem: do you know which boundary node you are hitting (IP address)? If you receive an erroneous response and the x-request-id header is present, could you share that request ID with us?

29

Hey @rbirkner

Sorry for the delayed response.

From Feb 7th, 7:17-7:31 UTC every single minute (16 straight minutes) we again saw a surge in unresponsive behavior from the boundary nodes again. Hopefully this information will help narrow things down.

These errors are coming from a standalone cron process that is using agent-js, in a us-west-2 data center, which is located in Oregon. Since I’m using agent-js, I’m not exactly sure how I’d extract HTTP headers from the request error :thinking:

@kpeacock the code I’m running is using 0.15.2 of agent-js, is there a recommended way to expose the error headers and request id from the agent?

I’d imagine given the origin of the request, it’s either hitting the boundary node in Seattle (1 node) or Palo Alto (2 nodes).

30

Hello @icme

During the time that you mention, we experienced extremely high-load on the boundary nodes that exhausted the boundary node resources.

We have been improving the performance of the boundary nodes and putting mitigations in place. This has helped us already to handle some surges with minimal interruptions (for example one yesterday and two earlier today). However, the third one today, was too much and lead to the problems you experienced. Sorry!

We are looking into what happened and how to handle it better. It just takes time!

1 Like