Getting ENOMEM on a local replica

Hello,
I’m getting this error on a local replica while calling an endpoint that deals work without allocating much memory :

thread '<unnamed>' panicked at 'called `Result::unwrap()` on an `Err` value: ENOMEM', memory_tracker/src/lib.rs:574:22

FYI I have enough RAM and hard drive space left (several 1-s of gigs).

I’m wondering what could be the issue ? @ulan @dsarlis

Using dfx 0.12.1
Linux kernel 5.15

Hmm, that is strange error indeed.

Based on the dfx version you use, the panic comes from an attempt to mprotect a memory region here. The man page has some information on when ENOMEM might be returned; besides not having enough memory which you said you have enough, the rest might point to some bug.

I’m not very familiar with the memory tracker myself, Ulan would be the expert on this, so probably best to wait for him to reply.

1 Like

I have no clue but, a quick search on the forum pointed me to this thread that list the same memory_tracker, so just in case it can help (because other thread was resolved): Introducing ic-nix, a ready made, cross-platform, and bleeding edge dev environment! - #4 by mbeekman

something with m1, nix and aarch64 but again, no clue :grin:

1 Like

Hi @dymayday. The error is caused by the Wasm execution hitting the maximum limit of virtual memory regions configured on your system. You can change the limit using this command:

sudo sysctl -w vm.max_map_count=2097152
2 Likes

More information on max_map_count: linux kernel - How much memory could vm use - Stack Overflow

Awesome thank you @ulan !
It worked perfectly :ok_hand:

That’s an interesting behavior that I’m interested to hear your opinion about as it happens when I mem:take or delete a lot of data from a struct stored in a thread_local!.

Would you happen to know what is happening under the hood for this to happen ?

My guess would be that these operations trigger a long chain of object drops that access a lot of memory. Internally that causes creation of virtual memory maps in the Wasm executor.

1 Like