Pocket-ic, issues after upgrade

i’m running into the following error when running tests with pocket-ic on the most recent versions;

After upgrading from;

dfx version: 0.26.0
ic-cdk: 0.17.1
pocket-ic: 0.7.0
pocket-ic-server: 0.8.0

to;

dfx version: 0.26.0
ic-cdk: 0.18.0
pocket-ic: 0.8.0
pocket-ic-server: 0.9.0 (also tried 0.8.0)
> also had to include the "ic-management-canister-types" crate

im getting the following error;

2025-04-28T20:50:19.099242Z  INFO pocket_ic_server: The PocketIC server is listening on port 61676
thread 'tokio-runtime-worker' panicked at rs/state_manager/src/lib.rs:1437:21:
Failed to load checkpoint @9200: /Users/rem.codes/Documents/rem.codes/toolkit/toolkit_services/src/test_helper/nns_state/checkpoints/00000000000023f0/canister_states/00000000000000000101: failed to deserialize canister_states[rwlgt-iiaaa-aaaaa-aaaaa-cai]::canister_state_bits: Value out of range for type OnLowWasmMemoryHookStatus: Unexpected value of status of on low wasm memory hook: Unspecified
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace
thread 'tokio-runtime-worker' panicked at rs/pocket_ic_server/src/state_api/state.rs:756:14:
Failed to create PocketIC instance: JoinError::Panic(Id(21), "Failed to load checkpoint @9200: /Users/rem.codes/Documents/rem.codes/toolkit/toolkit_services/src/test_helper/nns_state/checkpoints/00000000000023f0/canister_states/00000000000000000101: failed to deserialize canister_states[rwlgt-iiaaa-aaaaa-aaaaa-cai]::canister_state_bits: Value out of range for type OnLowWasmMemoryHookStatus: Unexpected value of status of on low wasm memory hook: Unspecified", ...)
test test_get_config ... FAILED

code

impl Context {
    pub fn new() -> Self {
        let nns_state_path = PathBuf::from(env::var("CARGO_MANIFEST_DIR").unwrap())
            .parent()
            .expect("Failed to get parent dir")
            .join("test_helper/nns_state");

        let owner_account = Account::from(Principal::from_text(OWNER_PRINCIPAL).unwrap());

        let default_install_settings: Option<CanisterSettings> = Some(CanisterSettings {
            controllers: Some(vec![owner_account.owner]),
            compute_allocation: None,
            memory_allocation: None,
            freezing_threshold: None,
            reserved_cycles_limit: None,
            log_visibility: None,
            wasm_memory_limit: None,
        });

        if !PathBuf::from(env::var("CARGO_MANIFEST_DIR").unwrap())
            .parent()
            .expect("Failed to get parent dir")
            .join("test_helper/nns_state")
            .exists()
        {
            panic!("NNS state not found. Please run `bash scripts/prepare_test.sh` to load the NNS state.");
        }

        let pic = PocketIcBuilder::new()
            .with_application_subnet()
            .with_nns_subnet()
            .with_nns_state(nns_state_path) // this first included state is the nns subnet id
            .build();

        let toolkit_services_canister =
            pic.create_canister_with_settings(None, default_install_settings.clone());

        pic.add_cycles(toolkit_services_canister, 2_000_000_000_000);

        let toolkit_services_wasm_bytes = include_bytes!("../../wasm/toolkit_services.wasm.gz");

        pic.install_canister(
            toolkit_services_canister,
            toolkit_services_wasm_bytes.to_vec(),
            encode_args(()).unwrap(),
            Some(owner_account.owner),
        );

        Context {
            pic,
            owner_account,
            toolkit_services_canister,
        }
    }

  pub fn toolkit_services_query<T: DeserializeOwned + CandidType>(
          &self,
          sender: Sender,
          method: &str,
          args: Option<Vec<u8>>,
      ) -> Result<T, String> {
          let args = args.unwrap_or(encode_args(()).unwrap());
          let res = self
              .pic
              .query_call(
                  self.toolkit_services_canister,
                  sender.principal(),
                  method,
                  args,
              )
              .expect("Failed to call canister");
  
          Decode!(res.as_slice(), T).map_err(|e| e.to_string())
      }
}

test;

#[test]
fn test_get_config() -> Result<(), String> {
    let context = Context::new();

    // fetch the config
    let config_result = context.toolkit_services_query::<CanisterResult<Config>>(
        Sender::Owner,
        "get_config",
        None,
    )?;

    println!("config_result: {:?}", config_result);
    assert!(config_result.is_ok());

    Ok(())
}

thanks in advance

I think the above was a combination of errors, i narrowed it down to

thread 'tokio-runtime-worker' panicked at rs/state_manager/src/lib.rs:1437:21:
Failed to load checkpoint @9200: /Users/rem.codes/Documents/rem.codes/toolkit/toolkit_services/src/test_helper/nns_state/checkpoints/00000000000023f0/canister_states/00000000000000000101: failed to deserialize canister_states[rwlgt-iiaaa-aaaaa-aaaaa-cai]::canister_state_bits: Value out of range for type OnLowWasmMemoryHookStatus: Unexpected value of status of on low wasm memory hook: Unspecified

which seems like an issue loading the NNS state

1 Like

Where did you get the NNS state from? Can you perhaps try with a newer one? It seems to be an old checkpoint that lacks a new field. Mainnet has been upgraded since quite a while ago to have this field.

1 Like

Yeah it was a while back, could you maybe point me to the docs to do it again?

1 Like

I’m unaware of public docs on how to retrieve the NNS state. Did someone from our side help you with this in the past?

Found it, but if you have non public docs it would be appreciated as wel :smiley:

1 Like

If you used this before and it worked, I think you can go ahead and use it again. If you follow the steps to redownload the NNS state (first part of the guide) it should be fine I think.

1 Like

I’ve updated the NNS state snapshot in this (by now merged) PR.

2 Likes

It seems that the guide is made easier, at first it was downloading the whole ic repo as the starting point. Appreciate the replies.

A bit offtopic, is there are a SNS subnet state with SNSes, or some test to deploy a SNS?

I was informed that another team member (@NathanosDev) has written this guide and I would recommend to follow it to get a newer snapshot of the state in case you notice any such errors with reading an older state (also in future scenarios).

I believe there’s some testing framework for SNSs, @mraszyk will know better. You can test deployment of an SNS afaik with it but I don’t know if you can retrieve the state from the SNS subnet (but also you shouldn’t need it to test your own SNS deployment).

Issue seems to be resolved, just need to make some additional changes as the default create_canister_with_settings now tries to install it on the NNS subnet.

Hi @rem.codes,

There are at least two ways to get an SNS instance in your testing environment. The one I recommend is documented here: ic/rs/sns/testing at b8ee431b0f5744724473092fac621311a74e47e2 · dfinity/ic · GitHub
The crate provides some functions for creating new SNSs and running swaps till the end using simulated participants, so you can get to any lifecycle stage you want.

The other solution is GitHub - dfinity/snsdemo: Developer focused SNS deployment demo, that would be better suited for testing the NNS dapp. It involves creating a subnet snapshot that could be loaded again, which is convenient if you want to mutate an SNS via a frontend and then reuse later on.

2 Likes

Thanks! i think the second is what i’m searching for at this point to test functionalities one existing SNSes for toolkit!

@dsarlis @mraszyk

So the tests work individually now but when running a cargo test to run them all they still error out, any idea what this could be?

running 8 tests
2025-04-29T08:50:19.719600Z  INFO pocket_ic_server: The PocketIC server is listening on port 51496
thread 'TipThread' panicked at rs/state_manager/src/tip.rs:680:9:
Failed to get MergeCandidateAndMetrics: File system error for file /Users/rem.codes/Documents/rem.codes/toolkit/toolkit_services/src/test_helper/nns_state/tip/canister_states/000000000000000f0101/00000000000009f6_0000_vmemory_0.overlay: Failed get existing file length: /Users/rem.codes/Documents/rem.codes/toolkit/toolkit_services/src/test_helper/nns_state/tip/canister_states/000000000000000f0101/00000000000009f6_0000_vmemory_0.overlay No such file or directory (os error 2)
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace
thread 'tokio-runtime-worker' panicked at rs/state_manager/src/lib.rs:1268:21:
failed to wait for TipHandler thread: RecvError
thread 'tokio-runtime-worker' panicked at rs/pocket_ic_server/src/state_api/state.rs:756:14:
Failed to create PocketIC instance: JoinError::Panic(Id(21), "failed to wait for TipHandler thread: RecvError", ...)
test test_start_recurring_payment_timer_on_update ... FAILED
thread 'tokio-runtime-worker' panicked at rs/state_manager/src/lib.rs:1293:39:
Failed to init state layout: IoError { path: "/Users/rem.codes/Documents/rem.codes/toolkit/toolkit_services/src/test_helper/nns_state/tip", message: "Unable to remove old tip. Tip could be inconsistent", io_err: Os { code: 66, kind: DirectoryNotEmpty, message: "Directory not empty" } }
thread 'tokio-runtime-worker' panicked at rs/pocket_ic_server/src/state_api/state.rs:756:14:
Failed to create PocketIC instance: JoinError::Panic(Id(25), "Failed to init state layout: IoError { path: \"/Users/rem.codes/Documents/rem.codes/toolkit/toolkit_services/src/test_helper/nns_state/tip\", message: \"Unable to remove old tip. Tip could be inconsistent\", io_err: Os { code: 66, kind: DirectoryNotEmpty, message: \"Directory not empty\" } }", ...)
test test_start_recurring_payment_timer_on_create ... FAILED
thread 'TipThread' panicked at rs/state_manager/src/tip.rs:337:37:
Failed to reset tip to height @2643: I/O error while accessing file /Users/rem.codes/Documents/rem.codes/toolkit/toolkit_services/src/test_helper/nns_state/tip: Failed to convert reset tip to checkpoint to /Users/rem.codes/Documents/rem.codes/toolkit/toolkit_services/src/test_helper/nns_state/checkpoints/0000000000000a53 (err kind: NotFound): No such file or directory (os error 2)
thread 'tokio-runtime-worker' panicked at rs/state_manager/src/lib.rs:1268:21:
failed to wait for TipHandler thread: RecvError
thread 'tokio-runtime-worker' panicked at rs/pocket_ic_server/src/state_api/state.rs:756:14:
Failed to create PocketIC instance: JoinError::Panic(Id(23), "failed to wait for TipHandler thread: RecvError", ...)
test test_update_recurring_payment ... FAILED
thread 'tokio-runtime-worker' panicked at rs/pocket_ic_server/src/pocket_ic.rs:617:18:
Failed to copy state directory: Os { code: 2, kind: NotFound, message: "No such file or directory" }
thread 'tokio-runtime-worker' panicked at rs/pocket_ic_server/src/state_api/state.rs:756:14:
Failed to create PocketIC instance: JoinError::Panic(Id(27), "Failed to copy state directory: Os { code: 2, kind: NotFound, message: \"No such file or directory\" }", ...)
test test_enable_payment_module ... FAILED
thread 'TipThread' panicked at rs/state_manager/src/tip.rs:337:37:
Failed to reset tip to height @2643: I/O error while accessing file /Users/rem.codes/Documents/rem.codes/toolkit/toolkit_services/src/test_helper/nns_state/tip: Failed to convert reset tip to checkpoint to /Users/rem.codes/Documents/rem.codes/toolkit/toolkit_services/src/test_helper/nns_state/checkpoints/0000000000000a53 (err kind: AlreadyExists): File exists (os error 17)
thread 'tokio-runtime-worker' panicked at rs/state_manager/src/lib.rs:1268:21:
failed to wait for TipHandler thread: RecvError
thread 'tokio-runtime-worker' panicked at rs/pocket_ic_server/src/state_api/state.rs:756:14:
Failed to create PocketIC instance: JoinError::Panic(Id(29), "failed to wait for TipHandler thread: RecvError", ...)
test test_create_recurring_payment ... FAILED
test test_invoke_inactive_payment_module ... ok
test test_disable_payment_module ... ok
test test_delete_recurring_payment ... ok

failures:

---- test_start_recurring_payment_timer_on_update stdout ----

thread 'test_start_recurring_payment_timer_on_update' panicked at /Users/rem.codes/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/pocket-ic-8.0.0/src/nonblocking.rs:209:14:
Failed to get result: reqwest::Error { kind: Request, url: "http://127.0.0.1:51496/instances", source: hyper_util::client::legacy::Error(SendRequest, hyper::Error(IncompleteMessage)) }
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace

---- test_start_recurring_payment_timer_on_create stdout ----

thread 'test_start_recurring_payment_timer_on_create' panicked at /Users/rem.codes/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/pocket-ic-8.0.0/src/nonblocking.rs:209:14:
Failed to get result: reqwest::Error { kind: Request, url: "http://127.0.0.1:51496/instances", source: hyper_util::client::legacy::Error(SendRequest, hyper::Error(IncompleteMessage)) }

---- test_update_recurring_payment stdout ----

thread 'test_update_recurring_payment' panicked at /Users/rem.codes/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/pocket-ic-8.0.0/src/nonblocking.rs:209:14:
Failed to get result: reqwest::Error { kind: Request, url: "http://127.0.0.1:51496/instances", source: hyper_util::client::legacy::Error(SendRequest, hyper::Error(IncompleteMessage)) }

---- test_enable_payment_module stdout ----

thread 'test_enable_payment_module' panicked at /Users/rem.codes/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/pocket-ic-8.0.0/src/nonblocking.rs:209:14:
Failed to get result: reqwest::Error { kind: Request, url: "http://127.0.0.1:51496/instances", source: hyper_util::client::legacy::Error(SendRequest, hyper::Error(IncompleteMessage)) }

---- test_create_recurring_payment stdout ----

thread 'test_create_recurring_payment' panicked at /Users/rem.codes/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/pocket-ic-8.0.0/src/nonblocking.rs:209:14:
Failed to get result: reqwest::Error { kind: Request, url: "http://127.0.0.1:51496/instances", source: hyper_util::client::legacy::Error(SendRequest, hyper::Error(IncompleteMessage)) }

How are you passing the state to your PocketIC instance? I suspect that you’re passing the same state into multiple PocketIC instances simultaneously which leads to data races.

I do a Context::new() for every test, which worked fine before :thinking:

impl Context {
    pub fn new() -> Self {
        let owner_account = Account::from(Principal::from_text(OWNER_PRINCIPAL).unwrap());

        let default_install_settings: Option<CanisterSettings> = Some(CanisterSettings {
            controllers: Some(vec![owner_account.owner]),
            compute_allocation: None,
            memory_allocation: None,
            freezing_threshold: None,
            reserved_cycles_limit: None,
            log_visibility: None,
            wasm_memory_limit: None,
        });

        if !PathBuf::from(env::var("CARGO_MANIFEST_DIR").unwrap())
            .parent()
            .expect("Failed to get parent dir")
            .join("test_helper/nns_state")
            .exists()
        {
            panic!("NNS state not found. Please run `bash scripts/prepare_test.sh` to load the NNS state.");
        }

        let nns_state_path = PathBuf::from(env::var("CARGO_MANIFEST_DIR").unwrap())
            .parent()
            .expect("Failed to get parent dir")
            .join("test_helper/nns_state");

        let pic = PocketIcBuilder::new()
            .with_nns_subnet()
            .with_nns_state(nns_state_path) // this first included state is the nns subnet id
            .with_application_subnet()
            .build();

        let toolkit_services_canister =
            pic.create_canister_with_settings(None, default_install_settings.clone());

        pic.add_cycles(toolkit_services_canister, 2_000_000_000_000);

        let toolkit_services_wasm_bytes = include_bytes!("../../wasm/toolkit_services.wasm.gz");

        pic.install_canister(
            toolkit_services_canister,
            toolkit_services_wasm_bytes.to_vec(),
            encode_args((owner_account.owner, owner_account.owner)).unwrap(),
            Some(owner_account.owner),
        );

        Context {
            pic,
            owner_account,
            toolkit_services_canister,
        }
    }
}
#[test]
fn test_enable_payment_module() -> Result<(), String> {
    let context = Context::new();

    // fetch the config
    let config_result = context.toolkit_services_query::<CanisterResult<Config>>(
        Sender::Owner,
        "get_config",
        None,
    )?;

    println!("config_result: {:?}", config_result);
    assert!(config_result.is_ok());

    let config = config_result.unwrap();
    assert!(config.modules.is_empty());

    // enable the payroll module
    let enable_result = context.toolkit_services_update::<CanisterResult<()>>(
        Sender::Owner,
        "enable_module",
        Some(encode_args((ModuleType::Payments,)).unwrap()),
    )?;

    println!("enable_result: {:?}", enable_result);
    assert!(enable_result.is_ok());

    // fetch the config again
    let config_result = context.toolkit_services_query::<CanisterResult<Config>>(
        Sender::Owner,
        "get_config",
        None,
    )?;

    println!("config_result: {:?}", config_result);
    assert!(config_result.is_ok());

    let config = config_result.unwrap();
    assert!(config.modules.contains(&ModuleType::Payments));
    Ok(())
}

These are not the latest versions: could you please bump to server v9.0.0 and library v8.0.0?

I am running the latest, i checked it with a self generated NNS state and the one you provided via the picjs repo

The crash has been fixed in this PR:

  • PocketIC server binary for linux is available for download here;
  • PocketIC server binary for macOS is available for download here.

A PocketIC fix release will follow shortly.

Thanks a lot, @rem.codes, for reporting the issue and (privately) sharing instructions how to reproduce!

3 Likes