Icx-proxy upgrade leads to 400 (Bad Request)

Developers
1

I tried to upgrade my docker solution to the last proposed release of the replica but faced an issue (thread). After some debugging, I figured out that the issue is the upgrade of icx-proxy - i.e., if I upgrade everything (replica, ic-starter, etc.) except icx-proxy, everything works as expected.

Therefore, I would be curious to know what breaking change was shipped in icx-proxy between commit 044cfd5147fc97d7e5a214966941b6580c325d72 and d966b2737ca75f1bfaa84f21e7f3f7c54b5d7f33 or just how to solve the upgrade issue?

Code: 400 (Bad Request)
Body: Could not find a canister id to forward to

This error happens when agent-js, more precisely the HttpAgent, tries to fetch the root key by perfoming a fetch on /api/v2/status.

1 Like
2

To be more precise:

3

Hey @peterparker

I haven’t looked in detail at the two commits, but I think it is related to our performance improvements to icx-proxy and the problem that it is being used in different places without any good test coverage. For now, you will (unfortunately) have to stay with the old commit (at least for icx-proxy).

Longer version:
End of last year and early this year, we had quite some events, where the boundary nodes reached their capacity and access to the IC was severly limited. As part of that, we started going through all boundary node components trying to eliminate bottlenecks and inefficiencies.

We also made some quick fixes to icx-proxy that lead to the errors you are observing. The problem for is that icx-proxy is used in several different places without having proper integration tests in place. Hence, we only notice later when something breaks (e.g., dfx upgraded months after the changes).

The team is currently working on the new boundary node architecture. As part of that, we also work on a replacement for icx-proxy, which we call ic-gateway. Hence, we decided to not fix icx-proxy and ask users to stay on an older commit. We only maintain icx-proxy at this point and don’t improve anything.

Once ic-gateway is ready, we encourage people to switch to that.

I hope that helps and you understand why we decided to not “fix” the changes for now.

1 Like
4

Thanks for the explanation, appreciated! It sure makes sense, and I’m more than fine with pinning icx-proxy in my Docker image.

It would probably be nice to add a notice about this deprecation e.g. at the top of the icx-proxy README.

1 Like