🧪 Announcing PicJS: TypeScript/JavaScript support for PocketIC!

skilesare · October 7, 2024, 7:05pm

Any updates yet on http_outcalls being supported and what we need to do to enable them? I’m currently getting

  console.error
    PocketIC server encountered an error BadIngressMessage("Failed to answer to ingress 0x0517d5c59f12c0a7edff0576abaed50f9f237363ca435e0520df6efe282960e8 after 100 rounds.")

      at intervalMs (node_modules/@hadronous/pic/src/http2-client.ts:144:19)
      at async runPoll (node_modules/@hadronous/pic/src/util/poll.ts:17:24)

When trying to interact with a local EVM RPC canister.

 "@hadronous/pic": "^0.9.0-b0",

NathanosDev · October 7, 2024, 11:20pm

I have a new beta version that’s using the latest PocketIC server that should work with HTTPS Outcalls now, but I haven’t actually tested it yet. I’ll work on some example project to test it in the coming days, but feel free to try it out yourself in the meantime, there’s no breaking changes except the required agent-js version, which is now ^1.4.0.

npm i -D @hadronous/pic@beta

skilesare · October 8, 2024, 3:33am

Probably need to loop @rvanasa in on this one. Using the new beta I’m getting the same issue.

I’m trying to call request on the evm rpc canister. I have a custom provider of

localProvider = await evm_fixture.actor.registerProvider({
      cyclesPerCall: 1000000000n,
      credentialPath: "",
      hostname: "127.0.0.1:8545",
      credentialHeaders: [],
      chainId : 31337n,
      cyclesPerMessageByte: 1000000n,
    });

I’m trying to validate my code against a locally running canister. And I’ve tried two different ways.

Option 1: I’ve set a custom provider to be used

Custom: {
                url: "http://127.0.0.1:8545",
                headers: [],
              }

This one fails fairly straightforwardly with the following error and I’m guessing that maybe there is a miscalculation in the EVMRpc canister for custom providers because I’m loading up the call to the rpc canister with:

Cycles.add<system>(state.cycleSettings.amountPerEthOwnerRequest); //set to 2_500_000_000_000
      let result = await rpcActor.request(rpc, json, 6000);

(It is possible I’m loading in the wrong third parameter…this response is rarely more than 500 bytes but I’m supplying more for safety.)

expect(received).toEqual(expected) // deep equality

    Expected: "0x3e9185d16a6a0857a2db4ddc2c56cea34baee322"
    Received: [[{"Err": {"RPC": {"Ethereum": {"HttpOutcallError": {"IcError": {"code": {"CanisterReject": null}, "message": "http_request request sent with 11_540_000 cycles, but 113_562_800 cycles are required."}}}}}}]]

…with a third param of 500 I get

Received: [[{"Err": {"RPC": {"Ethereum": {"HttpOutcallError": {"IcError": {"code": {"CanisterReject": null}, "message": "http_request request sent with 7_540_000 cycles, but 61_562_800 cycles are required."}}}}}}]]

So clearly the second param affects things, but I can’t tweak it to push more cycles on to its HTTP outcall.

Option 2:

I set up a custom provider by authing my self and then adding:

localProvider = await evm_fixture.actor.registerProvider({
      cyclesPerCall: 1000000000n,
      credentialPath: "",
      hostname: "127.0.0.1:8545",
      credentialHeaders: [],
      chainId : 31337n,
      cyclesPerMessageByte: 1000000n,
    });

I have some suspicions right off the bat here since the host doesn’t specify HTTP or HTTPs and I’m guessing my local rpc is only HTTP. But the error I get doesn’t make a ton of sense.

Basically I now call with

rpc:{
              Provider: 22n
            }

I seem to have gotten past my 'not enough cycles thread, but I get about 40 or so messages of:

 console.error
    PocketIC server encountered an error BadIngressMessage("Failed to answer to ingress 0x68e42066d75f343287588aac8e78d47f833583c4b3c081b42f782e42ee200bba after 100 rounds.")

      at intervalMs (node_modules/@hadronous/pic/src/http2-client.ts:144:19)
      at async Timeout.runPoll [as _onTimeout] (node_modules/@hadronous/pic/src/util/poll.ts:17:24)

And then the last of these messages has this extra “This is a bug” tag:

024-10-08T02:57:45.949188Z ERROR pocket_ic_server::state_api::state: The instance is deleted immediately after an operation. This is a bug!
  console.error
    PocketIC server encountered an error BadIngressMessage("Failed to answer to ingress 0xc04b678382eff65ebdae32793d0720b9d88594a2196806250ee5944c2e084d65 after 100 rounds.")

      at intervalMs (node_modules/@hadronous/pic/src/http2-client.ts:144:19)
      at async Timeout.runPoll [as _onTimeout] (node_modules/@hadronous/pic/src/util/poll.ts:17:24)

And then I get a bunch of:

console.error
    PocketIC server encountered an error UpdateError { message: "Instance was deleted" }

      at intervalMs (node_modules/@hadronous/pic/src/http2-client.ts:144:19)
      at async Timeout.runPoll [as _onTimeout] (node_modules/@hadronous/pic/src/util/poll.ts:17:24)

Maybe it is trying to make all the requests as if they came from different nodes? I’ve tried setting up the rpc with both a nodes_in_subnet parameter of 1 and 31.

Now one issue that I have here is that I don’t have any kind of special canisters set up like the nns or system subnets. Perhaps that is my issue and I need to pull in and have the state bootstrapped?

Unfortunately, I don’t have any way to make the evm rpc more chatty to know what it is and isn’t doing.

(I did just try to install an https proxy and proxy 443 to the local rpc host(hardhat) but I received all the same errors.

rvanasa · October 8, 2024, 4:23am

Thanks for the ping! Do you have a repository or branch we could use to repro this locally?

On the main branch of the EVM RPC canister, there is a new (soon to be released) demo flag in the install args which deactivates cycles payments. This might be usable as a workaround if needed.

I have some suspicions right off the bat here since the host doesn’t specify HTTP or HTTPs and I’m guessing my local rpc is only HTTP.

Yep, this is correct.

CC @gregory-demay @THLO in case anything else stands out about these errors.

NathanosDev · October 8, 2024, 9:16am

The likely cause of this is that the HTTPS Outcall response needs to be mocked OR the PocketIC server needs to be running in live mode. Neither are supported in PicJS yet. I had dangerously assumed the mocking was an optional feature but I just learned it’s required.

I’ll add support for mocking and have it ready hopefully tonight, but realistically probably tomorrow.

skilesare · October 8, 2024, 3:31pm

I will look for it. Is it the case that PocketIC won’t actually get a way to call an external https site? Can’t dfx do this and if so, does this make it harder for pocketIC to replace the dfx replica?

I guess since it is javascript I could proxy the site through the js mock?

skilesare · October 8, 2024, 3:41pm

No repo yet, but I’ll post my set up here…it isn’t long:

describe("test orchestrator", () => {
  beforeEach(async () => {
    pic = await PocketIc.create(process.env.PIC_URL, {
      
      /* nns: {
        state: {
          type: SubnetStateType.FromPath,
          path: NNS_STATE_PATH,
          subnetId: Principal.fromText(NNS_SUBNET_ID),
        }
      }, */

      processingTimeoutMs: 1000 * 60 * 5,
    } );

    //await pic.setTime(new Date(2024, 1, 30).getTime());
    await pic.setTime(new Date(2024, 7, 10, 17, 55,33).getTime());
    await pic.tick();
    await pic.tick();
    await pic.tick();
    await pic.tick();
    await pic.tick();
    await pic.advanceTime(1000 * 5);

    

    await pic.resetTime();
    await pic.tick();

    //const subnets = pic.getApplicationSubnets();
    console.log("setting up canisters");

    orch_fixture = await pic.setupCanister<OrchService>({
      //targetCanisterId: Principal.fromText("q26le-iqaaa-aaaam-actsa-cai"),
      sender: admin.getPrincipal(),
      idlFactory: orchestratorIDLFactory,
      wasm: orch_WASM_PATH,
      //targetSubnetId: subnets[0].id,
      arg: IDL.encode(orchestratorInit({IDL}), []),
    });

    await pic.tick();
    await pic.tick();
    await pic.tick();

    console.log("orch_fixture", orch_fixture);

    evm_fixture = await pic.setupCanister<EVMService>({
      //targetCanisterId: Principal.fromText("q26le-iqaaa-aaaam-actsa-cai"),
      sender: admin.getPrincipal(),
      idlFactory: evmIDLFactory,
      wasm: evm_WASM_PATH,
      //targetSubnetId: subnets[0].id,
      arg: IDL.encode(evmInit({IDL}), [{ nodesInSubnet : 31 }]),
    });

    console.log("evm_fixture", evm_fixture);

    await pic.tick();
    await pic.tick();
    await pic.tick();
    await pic.tick();
    await pic.tick();

    await evm_fixture.actor.setIdentity(admin);

    let authReq = await evm_fixture.actor.authorize(admin.getPrincipal(), {RegisterProvider: null});

    console.log("authReq", authReq);

    localProvider = await evm_fixture.actor.registerProvider({
      cyclesPerCall: 1000000000n,
      credentialPath: "",
      hostname: "localhost",
      credentialHeaders: [],
      chainId : 31337n,
      cyclesPerMessageByte: 1000000n,
    });

    console.log("localProvider", localProvider); 

    await pic.tick();
    await pic.tick();
    await pic.tick();
    await pic.tick();
    await pic.tick();


    //configure the orchestrator with rpc info
  });

it.only(`can get owner from other chain`, async () => {

    orch_fixture.actor.setIdentity(admin);

    await pic.tick();

    let configureResult = await orch_fixture.actor.configure(
      [{MapNetwork:{
        network: {Ethereum : [31337n]},
        action : { Add : null},
        service: {
          Ethereum: {
            canisterId: evm_fixture.canisterId,
            rpc:{
              Provider: 22n
              /* Custom: {
                url: "http://127.0.0.1:8545",
                headers: [],
              } */
            }
          }
        }
      }}]
    );

    console.log("configureResult", JSON.stringify(configureResult, replacer, 2));

    await pic.tick();
    await pic.tick();
    await pic.tick();
    await pic.tick();


    const foundOwner  = await orch_fixture.actor.get_remote_owner([
      {
        network: {Ethereum : [31337n]},
        contract: "0xCf7Ed3AccA5a467e9e704C703E8D87F634fB0Fc9",
        tokenId: 1n,
      }
    ]);

    await pic.tick();
    await pic.tick();
    await pic.tick();
    await pic.tick();
    await pic.tick();

    console.log("got", foundOwner);
    console.log("admin", admin.getPrincipal().toString());

    expect(foundOwner).toEqual("0x3e9185d16a6a0857a2db4ddc2c56cea34baee322");
  });

My orchestrator is just calling the setup rpc with:

let rpcActor: EVMRPC.Service = actor(Principal.toText(canisterId));

``

  let json = "{
    \"jsonrpc\": \"2.0\",
    \"method\": \"eth_call\",
    \"params\": [
      {
        \"to\": \"" # contract # "\", 
        \"data\": \"0x" # ethFunctionToHex(ethFn) # callData # "\" 
      },
      \"latest\" 
    ],
    \"id\": 1
  }";

  Cycles.add<system>(state.cycleSettings.amountPerEthOwnerRequest);
  let result = await rpcActor.request(rpc, json, 1000);

NathanosDev · October 8, 2024, 4:08pm

Some minor feedback on your code:

pic.setTime() accepts Date | Number, so you don’t need to call getTime():

await pic.setTime(new Date(2024, 7, 10, 17, 55, 33));

pic.tick() accepts a number to determine how many times it should tick so you can just do this:

await pic.tick(5);

skilesare · October 8, 2024, 4:12pm

The first is an artifact of a scenario where I had a state from dfx running for the NNS and I needed to hard set the date to avoid a bunch of trash going to the console(and I think there may have even been an error).

Second: Nice. Did not know that.

dfxjesse · October 8, 2024, 8:21pm

I followed this guide step by step and now while calling my beforeAll to just set up the pic instance I get a timeout?

  beforeAll(async () => {
    pic = await PocketIc.create(process.env.PIC_URL, {
      nns: {
        fromPath: NNS_STATE_PATH,
        subnetId: Principal.fromText(NNS_SUBNET_ID),
      },
    });
    await pic.setTime(new Date(2024, 10, 8).getTime());
    await pic.tick();
  });

Has anyone got this before? Not sure where to dig in deeper to find out what’s wrong here

NathanosDev · October 9, 2024, 9:41am

Can you try to enable additional logging and see if that gives any clues as to what’s going on? Running tests | PicJS (hadronous.github.io).

Possible reasons for this not working that I ran into before are:

The date being wrong. You can try set the date to be the day after the day that you created the state, just to be sure that it is strictly later than the creation date. Since new Date(2024, 10, 8) does not include a time component it would create a date with all time components set to 0, which will very likely result in a time earlier than the time the state was created if the date components match.
The state not being created properly. The final step of creating the state is super important:

Wait a few seconds to make sure there are no more logs from DFX coming in, and there is at least checkpoint in the .dfx/network/local/state/replicated_state/node-100/state/checkpoints folder, then stop DFX:

If enough time is not given for a checkpoint to be created then this can also break the setup.

dfxjesse · October 10, 2024, 11:19am

Thank you for this help @NathanosDev . I haven’t gotten it to work yet with my nns_state. But from the enabling the logs it seems it is to do with the checkpoint not being loaded in? Can you clarify what exactly this looks like in the folder:

Does it generate a checkpoint file or folder and then I know it’s worked successfully? Here is the structure of my checkpoints folder. I left it running for a few hours.

NathanosDev · October 10, 2024, 11:23am

I left it running for a few hours

Only a few seconds is enough, that checkpoint folder looks fine (although I’m not an expert on this).

The other potential issue is incompatibility between DFX versions and PocketIC. What version of PicJS are you using? I’ve been meaning to build out a compatibility table to avoid this issue.

dfxjesse · October 10, 2024, 11:26am

I am using

dfx 0.24.0
nns extension 0.4.5
@hadronous/pic 0.8.1

What versions did it work for you? I can try those

NathanosDev · October 10, 2024, 11:31am

Try the beta version:

npm i -D @hadronous/pic@beta

This worked with DFX 0.24.0 for me the other day.

dfxjesse · October 10, 2024, 12:02pm

Thanks for the help! It’s working now with the beta version

skilesare · October 16, 2024, 11:19pm

Looping back on this to see if the mock stuff is in and if will help with my issue.

icme · November 22, 2024, 6:43am

Just hit a bug!
ERROR pocket_ic_server::state_api::state: The instance is deleted immediately after an operation. This is a bug!

dfx version - 0.24.0
pic-js version - 0.10.0-b0

Not much of a trace to go off of. I added a few logs around it, but once before the bug error message appears I see a PocketIC server encountered an error UpdateError { message: "Instance was deleted" } error, and then it repeats infinitely after the bug error message.

 console.log
    after execute topups

      at Object.<anonymous> (cycleops/memory-notification.test.ts:125:15)

  console.log
    after advance time

      at Object.<anonymous> (cycleops/memory-notification.test.ts:128:15)

  console.log
    second to last test

      at Object.<anonymous> (cycleops/memory-notification.test.ts:132:15)

  console.error
    PocketIC server encountered an error UpdateError { message: "Instance was deleted" }

      at intervalMs (../../node_modules/@hadronous/pic/src/http2-client.ts:90:19)
      at async Timeout.runPoll [as _onTimeout] (../../node_modules/@hadronous/pic/src/util/poll.ts:17:24)

2024-11-22T06:36:49.577335Z ERROR pocket_ic_server::state_api::state: The instance is deleted immediately after an operation. This is a bug!
  console.error
    PocketIC server encountered an error UpdateError { message: "Instance was deleted" }

      at intervalMs (../../node_modules/@hadronous/pic/src/http2-client.ts:90:19)
      at async Timeout.runPoll [as _onTimeout] (../../node_modules/@hadronous/pic/src/util/poll.ts:17:24)

  console.error
    PocketIC server encountered an error UpdateError { message: "Instance was deleted" }

  ... this error then repeats forever.

@mraszyk another benefit of official pic-js support - more PocketIC users and testers to find these bugs

Edit: I’ve been able to zero in closer to the issue.

The test where it was failing (with the logging above)

it("executes topups successfully", async () => {
  console.log("second to last test");
  expect("ok" in executeTopupsResult).toBe(true);
});

Updating this async test (that really isn’t async) to a synchronous test (i.e. removing the async keyword) allows this test to pass, but then I get this fun error.

Jest did not exit one second after the test run has completed.

'This usually means that there are asynchronous operations that weren't stopped in your tests. Consider running Jest with `--detectOpenHandles` to troubleshoot this issue.
TypeError: fetch failed
    at node:internal/deps/undici/undici:13178:13
    at processTicksAndRejections (node:internal/process/task_queues:95:5)
    at async makeRequest (/Users/byronbecker/Workspace/Dfinity_Projects/mvp/node_modules/@hadronous/pic/src/http2-client.ts:59:19)
    at async intervalMs (/Users/byronbecker/Workspace/Dfinity_Projects/mvp/node_modules/@hadronous/pic/src/http2-client.ts:77:21)
    at async Timeout.runPoll [as _onTimeout] (/Users/byronbecker/Workspace/Dfinity_Projects/mvp/node_modules/@hadronous/pic/src/util/poll.ts:17:24) {
  [cause]: Error: connect ECONNREFUSED 127.0.0.1:52977
      at TCPConnectWrap.afterConnect [as oncomplete] (node:net:1607:16) {
    errno: -61,
    code: 'ECONNREFUSED',
    syscall: 'connect',
    address: '127.0.0.1',
    port: 52977
  }
}

For more context, I also spin up pic before every test and tear it down after every test.
Seems like if a test runs too quickly then it’s runs into an issue tearing down or spinning up again.

Edit: ran this test again and am back to the first error. But removing the short test completely gets rid of the error.

mraszyk · November 22, 2024, 7:39am

Thanks a lot for reporting! I believe this bug has been fixed by this commit - note the sentence “This PR also fixes a bug when deleting an instance: an instance should not be deleted if it is still busy with a computation.” in the PR description.

Let me try to provide more context on the bug: while an operation (e.g., a request you submitted using pic-js) is running on a PocketIC instance, the instance state is “Busy”. Before the above commit, deleting an instance would unconditionally replace its state with “Deleted”:

            let mut instance = instances[instance_id].lock().await;
            match std::mem::replace(&mut instance.state, InstanceState::Deleted) {
                InstanceState::Available(pocket_ic) => {
                    std::mem::drop(pocket_ic);
                    break;
                }
                InstanceState::Deleted => {
                    break;
                }
                InstanceState::Busy { .. } => {}
            }

and then once the running operation finishes, the instance state is already “Deleted” instead of “Busy” triggering the error message “The instance is deleted immediately after an operation.”.

The fix commit first checks if the instance is “Available” (in particular, not “Busy”) before settings its state to be “Deleted”:

            let mut instance = instances[instance_id].lock().await;
            match &instance.state {
                InstanceState::Available(_) => {
                    let _ = std::mem::replace(&mut instance.state, InstanceState::Deleted);
                    break;
                }
                InstanceState::Deleted => {
                    break;
                }
                InstanceState::Busy { .. } => {}
            }

icme · November 23, 2024, 6:15am

@skilesare helped me find the issue.

In case anyone has this issue in the future, I had a few pic.tick() and pic.advanceTime() calls that weren’t being awaited, so adding await before the command fixed the issue.

I’m guessing the tick/advanceTime might of been hanging while other state changes were trying to execute, or while I was trying to tear down the current pic instance.

Topic		Replies	Views
🤗 PocketIC: Fast and versatile Canister Testing in Rust and Python Developers	44	1988	May 6, 2024
There is a bug with @hadronous/pic Developers	4	73	November 23, 2024
PocketIC and PIC.js parallel behavior Language Support	19	185	January 14, 2025
Pocket-ic, issues after upgrade Developers pocketic	20	248	June 9, 2025
PocketIC: Multi-Subnet Canister Testing Developers rust	28	1511	June 29, 2024

🧪 Announcing PicJS: TypeScript/JavaScript support for PocketIC!

Related topics