Too many open files (os error 24)" })', state_manager/src/lib.rs:2943:33

Developers
1

While doing a demo of my new side project my local replica began to crash (:grimacing:) and the issue according stacktracke does not seem linked to my code. Any clue what’s happening and why I land in that state?

❯ dfx start
Running dfx start for version 0.12.1
Using project-specific network ‘local’ defined in /Users/daviddalbusco/projects/lab/poc/admin/dfx.json
WARN: Project-specific networks are deprecated and will be removed after February 2023.
Dashboard: http://localhost:58200/_/dashboard
Jan 27 09:52:08.807 CRIT s:krwzw-mh2yv-qldie-yvnu4-ex27s-j34lm-fhdit-glme3-6ywsj-yhe36-zae/n:kzt4d-utzl2-7lwci-l2waf-roe5n-j37yz-c2263-jh2eh-jcoi3-ml22c-6qe/ic_state_manager/ic_state_manager Failed to make a checkpoint @258200: Persistence(FileSystemError { path: “/Users/daviddalbusco/projects/lab/poc/admin/.dfx/network/local/state/replicated_state/node-100/state/checkpoints/000000000003f098/canister_states/00000000000000140101/stable_memory.bin”, context: “Failed to open file”, internal_error: “Too many open files (os error 24)” })
thread ‘MR Batch Processor’ panicked at ‘Failed to make a checkpoint @258200: Persistence(FileSystemError { path: “/Users/daviddalbusco/projects/lab/poc/admin/.dfx/network/local/state/replicated_state/node-100/state/checkpoints/000000000003f098/canister_states/00000000000000140101/stable_memory.bin”, context: “Failed to open file”, internal_error: “Too many open files (os error 24)” })’, state_manager/src/lib.rs:2943:33
note: run with RUST_BACKTRACE=1 environment variable to display a backtrace
Stopping icx-proxy…
Stopped.
Dashboard: http://localhost:58573/_/dashboard
Jan 27 09:54:22.334 CRIT s:krwzw-mh2yv-qldie-yvnu4-ex27s-j34lm-fhdit-glme3-6ywsj-yhe36-zae/n:kzt4d-utzl2-7lwci-l2waf-roe5n-j37yz-c2263-jh2eh-jcoi3-ml22c-6qe/ic_state_manager/ic_state_manager Failed to make a checkpoint @258400: Persistence(FileSystemError { path: “/Users/daviddalbusco/projects/lab/poc/admin/.dfx/network/local/state/replicated_state/node-100/state/checkpoints/000000000003f160/canister_states/000000000000001d0101/stable_memory.bin”, context: “Failed to open file”, internal_error: “Too many open files (os error 24)” })
thread ‘MR Batch Processor’ panicked at ‘Failed to make a checkpoint @258400: Persistence(FileSystemError { path: “/Users/daviddalbusco/projects/lab/poc/admin/.dfx/network/local/state/replicated_state/node-100/state/checkpoints/000000000003f160/canister_states/000000000000001d0101/stable_memory.bin”, context: “Failed to open file”, internal_error: “Too many open files (os error 24)” })’, state_manager/src/lib.rs:2943:33
note: run with RUST_BACKTRACE=1 environment variable to display a backtrace

1 Like
2

Gonna link also my other issue Dfx start endless loop - #2 by peterparker. Maybe something weird on M2?

3

internal_error: “Too many open files (os error 24)”

Seems like we need to increase the number of max open files in the OS:

launchctl limit maxfiles
sudo launchctl limit maxfiles <new soft limit> <new hard limit>
4

Thanks for the answer. I assume you would like me to run the command line right? So here the results:

Last login: Fri Jan 27 11:40:15 on ttys002
❯ launchctl limit maxfiles
maxfiles 256 unlimited

5

^^^ below is also the command to increase the limits.

1 Like
6

I’ll clean my local state and deploy again. If I hit the issue again, will increase the value, thanks for the answer!

1 Like
7

I’ve run into this regularly as well, but never had the problem before 0.12.1. I guess this due to a change/bug in dfx?

8

It’s a global limit on open files, and all the running applications and active users contribute there.

Also, the dfx might be using more open files if there are more running canisters. So it might be not the dfx per ce, but just the project got bigger…

Anyway, there is lsof command to check which files are currently open by which application/user. If the dfx is indeed doing something unexpected, please let us know.

1 Like
9

Hit this error again right when it should not have happened because I needed my current state to test a migration :smiling_face_with_tear:.

Anyhow, bumped the number of open files, same stuffs, does not work.

Had

❯ launchctl limit maxfiles
maxfiles 256 unlimited

Bumped to

❯ launchctl limit maxfiles
maxfiles 512 10240

Still got

thread ‘’ panicked at ‘Failed to start sandbox process: Os { code: 24, kind: Uncategorized, message: “Too many open files” }’, rs/canister_sandbox/replica_controller/src/launch_as_process.rs:146:6
note: run with RUST_BACKTRACE=1 environment variable to display a backtrace