Dfx local replica error reading a body from connection: end of file before message length reached

Developers
1

I recently upgraded to dfx v0.15.1 and after developing for a bit, the local replica began to throw following error:

2023-11-19T12:57:24.782864Z ERROR tower_http::trace::on_failure: response failed classification=Status code: 500 Internal Server Error latency=67806 ms
2023-11-19T12:57:24.783402Z ERROR icx_proxy_dev::proxy::agent: Unable to read body: Failed to read body: “error reading a body from connection: end of file before message length reached”
2023-11-19T12:57:24.783458Z ERROR tower_http::trace::on_failure: response failed classification=Status code: 500 Internal Server Error latency=67897 ms
2023-11-19T12:57:24.784083Z ERROR icx_proxy_dev::proxy::agent: Unable to read body: Failed to read body: “error reading a body from connection: end of file before message length reached”
2023-11-19T12:57:24.784118Z ERROR tower_http::trace::on_failure: response failed classification=Status code: 500 Internal Server Error latency=0 ms
2023-11-19T12:57:24.785606Z ERROR icx_proxy_dev::proxy::agent: Unable to read body: Failed to read body: “error reading a body from connection: end of file before message length reached”
2023-11-19T12:57:24.785646Z ERROR tower_http::trace::on_failure: response failed classification=Status code: 500 Internal Server Error latency=14722 ms
2023-11-19T12:57:24.785972Z ERROR icx_proxy_dev::proxy::agent: Unable to read body: Failed to read body: “error reading a body from connection: end of file before message length reached”
2023-11-19T12:57:24.785991Z ERROR tower_http::trace::on_failure: response failed classification=Status code: 500 Internal Server Error latency=68075 ms
2023-11-19T12:57:24.786502Z ERROR icx_proxy_dev::proxy::agent: Unable to read body: Failed to read body: “error reading a body from connection: end of file before message length reached”
2023-11-19T12:57:24.787030Z ERROR tower_http::trace::on_failure: response failed classification=Status code: 500 Internal Server Error latency=1 ms
2023-11-19T12:57:24.787116Z ERROR icx_proxy_dev::proxy::agent: Unable to read body: Failed to read body: “error reading a body from connection: end of file before message length reached”
2023-11-19T12:57:24.787130Z ERROR tower_http::trace::on_failure: response failed classification=Status code: 500 Internal Server Error latency=67990 ms
2023-11-19T12:57:24.786561Z ERROR icx_proxy_dev::proxy::agent: Unable to read body: Failed to read body: “error reading a body from connection: end of file before message length reached”
2023-11-19T12:57:24.787173Z ERROR tower_http::trace::on_failure: response failed classification=Status code: 500 Internal Server Error latency=1 ms
2023-11-19T12:57:24.786720Z ERROR icx_proxy_dev::proxy::agent: Unable to read body: Failed to read body: “error reading a body from connection: end of file before message length reached”
2023-11-19T12:57:24.787348Z ERROR tower_http::trace::on_failure: response failed classification=Status code: 500 Internal Server Error latency=68165 ms
2023-11-19T12:57:24.787515Z ERROR icx_proxy_dev::proxy::agent: Unable to read body: Failed to read body: “error reading a body from connection: end of file before message length reached”
2023-11-19T12:57:24.787529Z ERROR tower_http::trace::on_failure: response failed classification=Status code: 500 Internal Server Error latency=0 ms
2023-11-19 12:58:17.104856 UTC: [Canister qhbym-qaaaa-aaaaa-aaafq-cai] Indexed: 0 waiting : 60s
2023-11-19T12:58:23.250661Z ERROR tower_http::trace::on_failure: response failed classification=Status code: 500 Internal Server Error latency=89 ms

What’s that new error and why it is thrown?

2

I’m the only facing this error?

❯ dfx start
Running dfx start for version 0.15.1
Using project-specific network ‘local’ defined in /Users/daviddalbusco/projects/lab/emulator-nns/dfx.json
2023-11-26T09:39:47.877912Z ERROR tower_http::trace::on_failure: response failed classification=Status code: 503 Service Unavailable latency=2 ms
2023-11-26T09:39:47.880425Z ERROR tower_http::trace::on_failure: response failed classification=Status code: 503 Service Unavailable latency=0 ms
2023-11-26T09:39:47.882394Z ERROR tower_http::trace::on_failure: response failed classification=Status code: 503 Service Unavailable latency=0 ms
2023-11-26T09:39:47.886632Z ERROR tower_http::trace::on_failure: response failed classification=Status code: 503 Service Unavailable latency=0 ms
2023-11-26T09:39:49.449674Z ERROR tower_http::trace::on_failure: response failed classification=Status code: 503 Service Unavailable latency=0 ms
2023-11-26T09:39:49.450915Z ERROR tower_http::trace::on_failure: response failed classification=Status code: 503 Service Unavailable latency=0 ms
2023-11-26T09:39:49.452509Z ERROR tower_http::trace::on_failure: response failed classification=Status code: 503 Service Unavailable latency=0 ms
2023-11-26T09:39:49.454723Z ERROR tower_http::trace::on_failure: response failed classification=Status code: 503 Service Unavailable latency=0 ms
Initialized replica.
Dashboard: http://localhost:54753/_/dashboard

3

I ran dfx start --clean and it’s working fine

4

I’ve now encountered this issue in two separate projects within a week. I ran the --clean option less than an hour ago, and while removing the dfx cache resolved the issue, I’m curious to know if others are facing the same problem.

If no one else is experiencing it, then it’s probably a local issue on my machine, might good be.

However, if others are encountering this problem as well, it’s probably a bug in dfx that needs to be addressed.

5

I’m not sure, this might be a bug. However, I upgraded the dfx to 0.15.1 last month but didn’t encounter this particular error yet.

1 Like
6

I’m also getting
ERROR tower_http::trace::on_failure: response failed classification=Status code: 500 Internal Server Error when trying to fetch this resource:
bd3sg-teaaa-aaaaa-qaaba-cai.localhost:8080/.well-known/ii-alternative-origins

The funny thing is that it does load from my windows 11 browser without the error, but in chrome (test version 119.0.6045.105) on wsl2 Ubuntu, dfx throws the error in the terminal log

Dfx runs in wsl2 ubuntu

More details:
Chrome running in wsl actually requests *.icp0.io and is set to map all requests to a vite server also on wsl2 that handles tls and forwards to dfx at *.localhost

Curious to learn what it could be

1 Like
7

still getting
2023-11-29T14:30:52.021605Z ERROR tower_http::trace::on_failure: response failed classification=Status code: 500 Internal Server Error latency=4 ms

Thrown by dfx dependency? See setup described above

dfx version dfx 0.15.2-beta.3

8

On my side the error is random. I did not try the beta dfx though. Last few times I restarted my machine but, it still surfaced at some point.