Pocket-IC Integrations test failures

Developers Rust
1

I have some integrations tests which pass if I run it one by one but the moments I run all of them, half of the tests fail randomly with these different messages in different tests.

{ kind: Request, url: "http://127.0.0.1:58118/instances", source: hyper_util::client::legacy::Error(SendRequest, hyper::Error(IncompleteMessage)) }

Failed to create PocketIC instance: JoinError::Panic(Id(4063), "failed to spawn thread: Os { code: 35, kind: WouldBlock, message: \"Resource temporarily unavailable\" }", ...)
thread 'tokio-runtime-worker' panicked at rs/pocket_ic_server/src/state_api/state.rs:723:14:

Failed to create PocketIC instance: JoinError::Panic(Id(3965), "failed to spawn thread: Os { code: 35, kind: WouldBlock, message: \"Resource temporarily unavailable\" }", ...)

While I was not able to replicate on my machine; my colleague had some success with disabling any tests that uses PocketIC live mode but on my machine; that didn’t help at all.

DFX version - 0.24.3.

I also use the ulimit -n unlimited this didn’t seem to help

2

This happens if the degree of parallelism is too high and your machine runs out of resources. I’d suggest to find a sweet spot for --test-threads in your cargo test invokation.

3

I did try running with different values for the test-threads; values I choose 2, 4,8 but this didn’t seem to help.

4

Could you please share more details about your test setup and where the error is triggered?

1 Like
5

We currently have 80+ integration tests that use PocketIC. For isolation, each test spins up a fresh PocketIC instance to ensure a clean environment.

However, we’ve observed an inconsistent behavior: around 50% of the tests randomly pass, while the others fail. There’s no consistent failure point—tests might pass or fail at different stages across runs.

Once the first test panics, it seems to cascade—subsequent tests begin to fail more consistently, suggesting a possible side effect or shared resource issue despite the isolated instances.

Unfortunately, there’s no clear or repeatable location for the failure. The panic appears to be random, making debugging difficult.

6

I see the following two options given the above feedback:

  • use only one test thread (i.e., run all tests sequentially);
  • identify which test(s) are heavy, run those tests sequentially, and tune the number of test threads for the remaining (lighter) tests.
1 Like
7

I’m a bit confused by the above two pieces of information: if you have integration tests using PocketIC (in Rust), then I’d expect you use the pocket-ic (Rust) crate for those tests (and either download a compatible PocketIC server binary or rely on the pocket-ic crate to download one automatically).

8

Yes that is what am doing. Sorry for the confusion