PocketIC: Multi-Subnet Canister Testing

Developers
1

PocketIC: Multi-Subnet Canister Testing

Today, we released PocketIC v2.0.0, which enables canister developers to test in a multi-subnet setting. The new server version comes with matching updates to the Rust library and the Python library.

Multi-subnet testing is a novelty in the ecosystem, so consider this feature slightly experimental. Your feedback will help it mature!

What is PocketIC?

A fast and deterministic canister testing platform. Now with real cross-net calls!
For more general info, see the previous thread.

Multi-Subnet Testing: How?

Here are the key features of multi-subnet PocketIC, as exposed by the PocketIC Rust library:

Create an IC instance with an NNS subnet and two application subnets:

let pic = PocketIcBuilder::new()
    .with_nns_subnet()
    .with_application_subnet()
    .with_application_subnet()
    .build();

Target the NNS subnet specifically to create a canister:

let nns_sub = pic.topology().get_nns_subnet().unwrap();
let nns_can_id = pic.create_canister_on_subnet(..., nns_sub);

Target one of the app subnets specifically to install a canister:

let app_sub_2 = pic.topology().get_app_subnets()[1];
let app_can_id = pic.create_canister_on_subnet(..., app_sub_2);

Create a canister with a specific canister_id on a named subnet

I.e., the subnet with the corresponding canister_id range; in this example, the NNS subnet:

let ledger_canister_id = Principal::from_text("ryjl3-tyaaa-aaaaa-aaaba-cai").unwrap();
pic.create_canister_with_id(..., ledger_canister_id);
pic.install_canister(ledger_canister_id, ...);

The subnet types include:

  • Generic system subnet
  • Generic application subnet
  • Named subnet with canister_id range like on mainnet: NNS, SNS, II, Fiduciary, Bitcoin. Only these can be targeted by install_canister_with_id()

What Breaking Changes Come With v2.0.0?

Very few.

In the Rust client library API:

  • create_canister() takes no arguments; to specify a sender, use create_canister_with_settings()
  • call_candid_as() takes an additional argument Option<EffectivePrincipal>

A notable difference is that by default, PocketIC used to contain a system subnet, and from v2.0.0 onwards, it defaults to containing an application subnet. Because the message size limits are now smaller by default, some calls with large payloads could now fail. Note, however, that this is more realistic, because on mainnet you install on an app subnet anyway.

Calling Tick on an IC instance now attempts to make progress on all subnets. This is necessary because that is the only way potential cross-net messages can be processed. Therefore, be extra careful with timers/heartbeats, which are and always have been only tentatively supported by PocketIC.

For details, see the changelogs of the PocketIC server and of the Rust library.

Questions?

Please reach out to us in this thread. We are happy to help with your setup, and are keen to learn about your bug reports and feature requests.

19 Likes
2

After upgrading from 1.0, many of our tests now fail with this error:

UserError(UserError { code: CanisterInstallCodeRateLimited, description: "Canister lxzze-o7777-77777-aaaaa-cai is rate limited because it executed too many instructions in the previous install_code messages. Please retry installation after several minutes." })

any advice?

3

Hi @ufoscout, thanks for reaching out.
This seems to be related to the fact that version 2.0 defaults to an application subnet to more closely resemble the mainnet. Version 1.0 was using a system subnet by default.
On mainnet, your calls would likely also fail with that error message; may I ask why youā€™re executing these calls?
One way you can circumvent this is to call pic.advance_time(X minutes), execute a few rounds with pic.tick() and retry.
Does this help?

4

Adding to fxgstā€™s reply, when installing large canisters, the canister can get into temporary instruction (think cycles) debit and refuse to accept new install_code messages. The debit from an install_code cannot exceed 200B instructions, and every empty round, 2B instructions will be taken from any debit. So a few empty ticks, at most 100, should suffice.

advance_time should not help.

1 Like
5

Thanks, this solved the issue:

    for _ in 0..100 {
        pocket_ic.tick();
    }
6

Another question: we have some integration tests that, through reqwest, make an HTTP call to the http_request endpoint and verify that the call is upgraded to an http_request_update. Is there a way to test this scenario with PocketIC?"

7

Unfortunately making HTTP requests is not supported yet. For this to work, an HTTP Gateway needs to be integrated into PocketIC. Weā€™ve already been discussing this week the best way to do this. Thereā€™s no timeline yet, but thereā€™ll be an update here as soon as thereā€™s some progress there! In the meantime youā€™ll need to continue testing these kinds of scenarios via DFX.

EDIT: Alternatively, you can call http_request endpoint of your canister through the PocketIC interface instead of with reqwest and assert that the response has the upgrade flag set to true. If you do it this way, then youā€™ll need to manually call the http_request_update endpoint if you want to continue.

3 Likes
8

@NathanosDev thanks for the clarification

9

Is there a way to disable the canister logs (the ones generated by print)? On v1 they were disabled by default I think, while on v2 they are printed in the stdout of the terminal in which I run cargo test.

Also, Iā€™m wondering whatā€™s causing the following log, if I should solve it and how:

WARN s:/n:/ic_state_manager/ic_state_manager No state available with certification.
10

In my case, Iā€™m executing it at the start of most of the test functions in ic-websocket-cdk-rs integration tests, so that I can test each method of the CDK separately. So, I basically use the reinstall_canister function as a way to reset the canisterā€™s state.
Is there a better (and faster, since adding the v2 ticks fix makes the tests last for long) way to do it? Should I change the way I test the CDK instead?

11

Hi @ilbert, to mute all messages from the server, you can set the env variable POCKET_IC_MUTE_SERVER=1.
The ā€œNo state available with certificationā€ can safely be ignored. It is printed by the state machine when executing the first round since we donā€™t certify states for testing.

1 Like
12

I think you could also create a new canister in order to reset itā€™s state. I believe this should be faster than deleting, ticking and reinstalling it, but I havenā€™t tested it myself.

1 Like
13

This doesnā€™t make much difference for my tests, so I opted to another solution: using a custom wipe method on the canister to reset its internal state. This way I use the the same canister across all tests and I donā€™t have to install a new one each time.

1 Like
14

Iā€™m glad you figured it out!

1 Like
15

Is there a reason why now timers are not working anymore only in a Motoko canister?

Should I add more ticks?

16

Hi ilbert,

Are you saying your timers have worked before and now donā€™t?

17

Yes, thatā€™s the behavior Iā€™ve observed. Adding 100 ticks (instead of just 1 as I was doing with PocketIC v1) after advance_time seems to solve it.

18

If there are crossnet calls involved, I may have an explanation.

Think of a tick as a round of ticks for every subnet. The round always happens in the same order, so if you have an xnet call from a canister in subnet 2 to a canister in subnet 1, then the xnet call will take two rounds to be processed (whereas an xnet call from 1 to 2 only takes one round). We have some ideas on how to optimize this eventually.

Otherwise it may be due to the smaller default limits discussed above.

In any case, since PocketIC is deterministic, you may want to figure out how many ticks you really need here, because it could turn out to be just 3, 10 or so. The number must be stable across runs of the same wasm versions. A quick binary search could help.

1 Like
19

There are no crossnet calls involved, I just had to add the NNS subnet in order to be able to call the root_key method. The canister Iā€™m testing is anyway deployed on an app subnet.

Ok thanks! Iā€™ll stick to 100 for now as it seems to work and doesnā€™t affect the total time it takes to run all tests, but Iā€™ll keep your suggestion in mind.

20

@michael-weigelt I am facing this issue while testing multi-subnet using PocketIC. Testing Creation of Canister on a subnet using Cycle Minting Canister with PocketIC