🤗 *PocketIC*: Fast and versatile Canister Testing in Rust and Python

:tada: Today, we are excited to announce the official release of a new version of the Rust- and Python-PocketIC testing library alongside a new version of the PocketIC-Server. :tada:

PocketIC is a canister testing platform that supports deterministic, programmatic canister testing. It is lightweight and integrates seamlessly into existing testing infrastructure (e.g. cargo test). It runs as a standalone binary on MacOS and Linux without requiring additional containers or Virtual Machines. All that is required is that this binary be present on your host system.

Example:

use candid::{encode_one, Principal};
use pocket_ic::PocketIc;

 #[test]
 fn test_counter_canister() {
    let pic = PocketIc::new();
    // Create an empty canister as the anonymous principal.
    let canister_id = pic.create_canister(None);
    pic.add_cycles(canister_id, 1_000_000_000_000_000);
    let wasm_bytes = load_counter_wasm(...);
    pic.install_canister(canister_id, wasm_bytes, vec![], None);
    // 'inc' is a counter canister method.
    call_counter_canister(&pic, canister_id, "inc");
    // Check if it had the desired effect.
    let reply = call_counter_canister(&pic, canister_id, "read");
    assert_eq!(reply, WasmResult::Reply(vec![0, 0, 0, 1]));
 }

fn call_counter_canister(pic: &PocketIc, canister_id: Principal, method: &str) -> WasmResult {
    pic.update_call(canister_id, Principal::anonymous(), method, encode_one(()).unwrap())
        .expect("Failed to call counter canister")
}

In principle, it can support any programming language. It achieves that by providing a REST-API to interact with a subset of deterministic replica-components (the «Execution Environment»).

Our vision is that PocketIC replaces all current canister testing methods.

:thinking: Why PocketIC?

  • With PocketIC, test execution is reproducible, as non-deterministic components of the replica (such as consensus or networking) are replaced with deterministic counterparts.
  • PocketIC allows for fast test execution, as networking related timeouts are removed.
  • PocketIC is versatile and allows for more fine-grained control over the execution environment. For example, already today one can directly manipulate the stable memory of a canister and thus test canister upgrades.

:magic_wand: What’s next?

Support for multiple subnets

The next big feature which is planned for November is support for multi-subnet testing: As a test author, you will be able to define a topology and test interaction between canisters installed on different subnets. Also, the cycles accounting should scale with the subnet size, similar to mainnet. In our estimate, this is useful in particular—but not only—for testing scenarios that involve the SNS.

HTTP Outcalls

As a test author, you will be able to mock HTTP-responses in a test. Thus, you can test scenarios involving HTTP-outcalls deterministically.

Minor Improvements & Optimizations

We are constantly on the lookout for ways to provide more power to the test author without hurting the consistency of the API. For example, in the current API every ingress message is executed in a separate block. In the future, we will provide an API that allows pooling ingress messages and executing them within a single block. This is not only closer to the scenario in production, but should also speed up tests.

:hugs: Questions?

Reach out to us in the forum or leave a post here! We are happy to answer questions and help with the setup!

23 Likes

Hi, Nice crate, but how is this different from using state machine tests with the client, is there some advantages using this ? Thanks

2 Likes

For a test author in Rust, the experience was intentionally left the same, but we have made some changes under the hood:
The IC instances run on a local server, not as one process per test thread. PocketIC now uses a local server with an HTTP/JSON API, so it supports any client library language. In particular, this has enabled our new Python testing library for testing canister.
Also, it is now possible for the community to take on the task of developing a JS, TS, C++, … client library for PocketIC without depending on components from the IC repository, and we are currently exploring new features that are not possible with the StateMachine at the moment.

8 Likes

Today I’ve finished migrating tests that I wrote for a small side project over to PocketIC and the difference in test execution time is outrageous!

Previously I was running DFX, deploying a canister there and then making calls against that canister using agent-js. There’s about 10 tests here that each make multiple calls to the canister and they took 53 seconds to run. After migrating to PocketIC, those same tests are running in less than 2 seconds!

Additionally, because deploying the canister with DFX was slow, I only deployed the canister once. This meant that canister state was persisting between tests and I had to be very careful to not use the same data in multiple tests. So I had some functions to generate random data for each test. Now with PocketIC I’m able to deploy a new canister and get fresh state for every test and I can get rid of all this code for generating random data.

This is really a huge improvement in developer experience when testing canisters.

For those interested, I’ll be publishing the TypeScript client that I’m using in the coming weeks once I finish polishing it off!

12 Likes

Thanks for sharing. It is good to see just how much difference this can make with real projects.

It is also good to know that the REST API has already paid off, because it enabled you to make progress while we were still finalizing the Rust library.

2 Likes

Is it possible with PocketIC to simulate responses? For example say the CUT (canister under test) makes a call. Can I simulate that it gets a certain system error back? Or say it makes two calls. Can I control the order in which the responses come back?

4 Likes

Very cool!

Does the code run the same in pocketIC as in the dfx local network and in the main network?

In icpp, the C++ CDK, you can currently run smoke tests against canisters deployed to either the local network or the main network by writing pytest tests that import a few predefined fixtures, and then passing the appropriate value for --network=[local/ic] to the pytest command.

I am interested to explore the benefits of adding pocketIC as an option too, perhaps by adding an additional predefined pytest fixtures. What would be the main benefit? Is it the speed?

1 Like

Hi @icpp, PocketIC uses a different technology stack than the local replica. It tries to be as close to mainnet as possible, and it actually uses a lot of components from there.
You’re right, for various applications, we saw great speed improvements over other solutions, so I would say that is the main benefit. In addition, we’re exploring the integration of other features like support for multi-subnet in PocketIC.

1 Like

Hi @timo, at the moment, PocketIC does not have these capabilities. It tries to be as close to mainnet as possible (while still being deterministic), so to get a system error, you would have to actually send a message that would be rejected by the IC. Further, the message ordering is currently fixed.

The canister code runs on exactly the same environment as on the mainnet. We use the mainnet execution environment, just like the local replica does.

The speed improvements compared to testing on the replica can be significant, as Nathan shows above.

2 Likes

I used the PocketIC for some quick tests but it seems that it bypasses the inspect message checks. Am I doing something or it doesn’t support inspect message checks ?

inspect_message checks will be supported in a next release of pocketIC

2 Likes

The next release should be in a week or two from here

3 Likes

Thanks guy! appreciate the quick response. Looking forward to this release

1 Like

How can I control canisters timeout and timers on PocketIC? Should I just advance the time to make them expire?

By the way, is there an API specification of PocketIC?

Calling advance_time() and then tick() (otherwise PocketIC won’t produce a new block) should do the job. I have an example of that here, it’s in TypeScript but should translate easily to your Rust tests.

3 Likes

Hi @ilbert, regarding the API specification: we’ve tried to document all public functions of the Rust and Python libraries with docstrings. As for the REST API of the PocketIC server, we do not have a specification at the moment, but we are looking into adding one, as this has been requested by others as well.

1 Like

Just a quick question regarding this instantiation, does this spin up isolated execution environments/sandboxes for the separate integration tests that we run or are all of the canisters created running in the same execution environment and they share the environment.

The reason for the question is if they are separate sandboxes, I could have multiple integration tests being run by Cargo in parallel without them stepping on each other and this is the behaviour of state-machine-tests currently and it keeps things consistent and simple.

Thoughts?

2 Likes

Hi @saikatdas0790, what you’ve said is exactly right.
This line does the following:
It starts a PocketIC server if it’s not already running, otherwise the server is reused. Then, a fresh and independent instance (think StateMachine) is created, which you can use in your test. You can then install canisters on this instance, make calls etc. You can have multiple such instances running in parallel, so it works well with Cargo. The instances do not influence each other, and you don’t see canisters of other instances. When pic goes out of scope, the instance is automatically collected by the server. To see an example, please take a look at this file. Please let us know if you have any further questions!

3 Likes

It is pretty fast. Also when I try to push dfx (artificial delay 0) and populate the backend. I am getting Server returned an error: Code: 429 (Too Many Requests) Body: The service is overloaded.
While pocket-ic doesn’t complain.

Is it possible to add some of these optimisations to dfx, like

dfx start --clean --fast

This way a full stack dev can also see how their frontend looks like after running the tests and populating canisters with data & various test scenarios.

2 Likes