While doing a demo of my new side project my local replica began to crash () and the issue according stacktracke does not seem linked to my code. Any clue what’s happening and why I land in that state?
❯ dfx start
Running dfx start for version 0.12.1
Using project-specific network ‘local’ defined in /Users/daviddalbusco/projects/lab/poc/admin/dfx.json
WARN: Project-specific networks are deprecated and will be removed after February 2023.
Dashboard: http://localhost:58200/_/dashboard
Jan 27 09:52:08.807 CRIT s:krwzw-mh2yv-qldie-yvnu4-ex27s-j34lm-fhdit-glme3-6ywsj-yhe36-zae/n:kzt4d-utzl2-7lwci-l2waf-roe5n-j37yz-c2263-jh2eh-jcoi3-ml22c-6qe/ic_state_manager/ic_state_manager Failed to make a checkpoint @258200: Persistence(FileSystemError { path: “/Users/daviddalbusco/projects/lab/poc/admin/.dfx/network/local/state/replicated_state/node-100/state/checkpoints/000000000003f098/canister_states/00000000000000140101/stable_memory.bin”, context: “Failed to open file”, internal_error: “Too many open files (os error 24)” })
thread ‘MR Batch Processor’ panicked at ‘Failed to make a checkpoint @258200: Persistence(FileSystemError { path: “/Users/daviddalbusco/projects/lab/poc/admin/.dfx/network/local/state/replicated_state/node-100/state/checkpoints/000000000003f098/canister_states/00000000000000140101/stable_memory.bin”, context: “Failed to open file”, internal_error: “Too many open files (os error 24)” })’, state_manager/src/lib.rs:2943:33
note: run with RUST_BACKTRACE=1 environment variable to display a backtrace
Stopping icx-proxy…
Stopped.
Dashboard: http://localhost:58573/_/dashboard
Jan 27 09:54:22.334 CRIT s:krwzw-mh2yv-qldie-yvnu4-ex27s-j34lm-fhdit-glme3-6ywsj-yhe36-zae/n:kzt4d-utzl2-7lwci-l2waf-roe5n-j37yz-c2263-jh2eh-jcoi3-ml22c-6qe/ic_state_manager/ic_state_manager Failed to make a checkpoint @258400: Persistence(FileSystemError { path: “/Users/daviddalbusco/projects/lab/poc/admin/.dfx/network/local/state/replicated_state/node-100/state/checkpoints/000000000003f160/canister_states/000000000000001d0101/stable_memory.bin”, context: “Failed to open file”, internal_error: “Too many open files (os error 24)” })
thread ‘MR Batch Processor’ panicked at ‘Failed to make a checkpoint @258400: Persistence(FileSystemError { path: “/Users/daviddalbusco/projects/lab/poc/admin/.dfx/network/local/state/replicated_state/node-100/state/checkpoints/000000000003f160/canister_states/000000000000001d0101/stable_memory.bin”, context: “Failed to open file”, internal_error: “Too many open files (os error 24)” })’, state_manager/src/lib.rs:2943:33
note: run with RUST_BACKTRACE=1 environment variable to display a backtrace
It’s a global limit on open files, and all the running applications and active users contribute there.
Also, the dfx might be using more open files if there are more running canisters. So it might be not the dfx per ce, but just the project got bigger…
Anyway, there is lsof command to check which files are currently open by which application/user. If the dfx is indeed doing something unexpected, please let us know.
Hit this error again right when it should not have happened because I needed my current state to test a migration .
Anyhow, bumped the number of open files, same stuffs, does not work.
Had
❯ launchctl limit maxfiles
maxfiles 256 unlimited
Bumped to
❯ launchctl limit maxfiles
maxfiles 512 10240
Still got
thread ‘’ panicked at ‘Failed to start sandbox process: Os { code: 24, kind: Uncategorized, message: “Too many open files” }’, rs/canister_sandbox/replica_controller/src/launch_as_process.rs:146:6
note: run with RUST_BACKTRACE=1 environment variable to display a backtrace
I removed this whole dir here File system error for file /Users/cyberowl/Library/Application Support/org.dfinity.dfx and then did sh -ci "$(curl -fsSL https://smartcontracts.org/install.sh)". Moved it to desktop in case I need it but so far seems fine.
Too many open files (os error 24) … here we go again
Failed to get existing overlays for StableMemory(
CanisterId(
xaonm-oiaaa-aaaaq-aabgq-cai,
),
): I/O error while accessing file /Users/daviddalbusco/projects/dfinity/oisy-wallet/.dfx/network/local/state/replicated_state/node-100/state/tip/canister_states/000000000200004d0101: Failed list overlays: Too many open files (os error 24)
Apr 16 06:51:39.524 CRIT s:da2wg-kvuzu-6oe54-yudvh-r4cyx-xwpei-al5b5-pddtq-y6uvc-vo4e6-tae/n:rwuu4-npotj-rzxvn-fkcnh-kpydf-i6yxp-id4x4-i2ufm-kvtez-ijtai-kae/ic_state_manager/tip Failed to get existing overlays for StableMemory(
CanisterId(
7vojr-tyaaa-aaaaq-aaatq-cai,
),
): I/O error while accessing file /Users/daviddalbusco/projects/dfinity/oisy-wallet/.dfx/network/local/state/replicated_state/node-100/state/tip/canister_states/00000000020000270101: Failed list overlays: Too many open files (os error 24)
Stopping icx-proxy...
Stopped.
2024-04-16T06:51:46.162594Z ERROR tower_http::trace::on_failure: response failed classification=Status code: 503 Service Unavailable latency=1 ms
2024-04-16T06:51:46.173092Z ERROR tower_http::trace::on_failure: response failed classification=Status code: 503 Service Unavailable latency=1 ms
2024-04-16T06:51:46.178723Z ERROR tower_http::trace::on_failure: response failed classification=Status code: 503 Service Unavailable latency=0 ms
2024-04-16T06:51:46.185993Z ERROR tower_http::trace::on_failure: response failed classification=Status code: 503 Service Unavailable latency=1 ms
or
Apr 16 07:04:25.979 CRIT s:o62sa-vmyf2-bdwh2-27tlc-o4pq7-ugpfa-kax7t-h3f4z-5hdbx-ssoc4-tqe/n:qzeew-3ojii-yoywt-ir2ym-apnch-ucjne-ti7by-fi72p-pam66-qvlwo-tae/ic_state_manager/tip Failed to persist unflushed delta: File system error for file /Users/daviddalbusco/projects/dfinity/oisy-wallet/.dfx/network/local/state/replicated_state/node-100/state/tip/canister_states/000000000200005c0101/stable_memory.bin: Failed to open file Too many open files (os error 24)
thread 'Apr 16 07:04:25.980 CRIT s:o62sa-vmyf2-bdwh2-27tlc-o4pq7-ugpfa-kax7t-h3f4z-5hdbx-ssoc4-tqe/n:qzeew-3ojii-yoywt-ir2ym-apnch-ucjne-ti7by-fi72p-pam66-qvlwo-tae/ic_state_manager/tip Failed to persist unflushed delta: File system error for file /Users/daviddalbusco/projects/dfinity/oisy-wallet/.dfx/network/local/state/replicated_state/node-100/state/tip/canister_states/00000000020000520101/stable_memory.bin: Failed to open file Too many open files (os error 24)
<unnamed>' panicked at rs/state_manager/src/tip.rs:340:49:
Failed to persist unflushed delta: File system error for file /Users/daviddalbusco/projects/dfinity/oisy-wallet/.dfx/network/local/state/replicated_state/node-100/state/tip/canister_states/000000000200005c0101/stable_memory.bin: Failed to open file Too many open files (os error 24)
thread 'note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace
<unnamed>' panicked at rs/state_manager/src/tip.rs:340:49:
Failed to persist unflushed delta: File system error for file /Users/daviddalbusco/projects/dfinity/oisy-wallet/.dfx/network/local/state/replicated_state/node-100/state/tip/canister_states/00000000020000520101/stable_memory.bin: Failed to open file Too many open files (os error 24)
This time got the feeling I landed here because I made too many installation of canisters in parallel.
) and the issue according stacktracke does not seem linked to my code. Any clue what’s happening and why I land in that state?
.