Future of the ic-ref emulator (dfx start --emulator)

I’d like to start a discussion with the community and the stakeholders at DFINITY about the use of ic-ref as an IC emulator in dfx.

Background on ic-ref

In the wake of introducing the Interface Specification, which defines the externally visible behaviour of the Internet Computer, we created a reference implementation called ic-ref. This is a completely independent implementation of the Interface Specification, and helped us to give the Spec more weight and maturity, by showing that it can be implemented. It also helped with testing of the agents and CDKs: By running their test suite against both production and reference implementation of the IC, when things don’t work, the cause can be triangulated more easily. Finally, it was (or was meant to be) a way to experiment with new features, and allow agents and CDKs to work on supporting them even before the replica has them.

ic-ref as a dev tool

As a reference implementation, ic-ref takes many shortcuts: Not optimized for performance, not networked, not necessarily secure, no disk usage. Some of these properties make it actually a nice way to run canisters locally, so a while ago, we started shipping ic-ref with dfx, and you can run your program against it instead of the production replica by running dfx start --emulator.

We also added features to ic-ref that are needed particularly for its use as a dev emulator, in particular the ability to dump and restore the state, so that you can install your canisters, stop the emulator, start it later, and don’t have to re-install them.

There were bolder visions, of course: A self-contained, simpler implementation of the IC with a Wasm interpreter could be extended with all kinds of cool development features that we may not want in the real replica: Better logging, live detailed debugging, manual control over message scheduling, manual insertion of faults etc.

But resources are limited, some of the poeple excited about this kind of work left or were made to leave the company shortly after I left, so these extra features did not happen.

Also, there were ideas of writing a fully dedicated dev emulator separate from ic-ref, which made it unclear if we should add them to ic-ref (but these also have moved on).

The cost of status quo

So now we are in the unfortunate state that ic-ref is shipped as the emulator, but doesn’t provide that much added value. At the same time it adds cost and friction:

  • The features that were added only for the emulator use case (mostly state serialization) need to be maintained, slow down building etc. Removing unused code is always good.
  • Building ic-ref in a way that an be distributed in a self-contained, distibution-agnostic way for Linux and MacOS is relatively tricky (and currently prevents a dependency bump in the repo, which is why I am writing this now, to procrastinate from debugging this issue).
  • The dfx release process is tied to yet another repository and toolchain, which adds complexity.
  • As far as I know there are currently no plans to add more dev-focused features to ic-ref, which would require more dev resources.

The questions

So, is having ic-ref shipped with dfx useful to anyone? Has anyone here ever used it for good effect? Or can we just remove this? Is this question worth an NNS motion?

(I am not questioning the use of ic-ref as the reference implementation, nor its use in the test suite of agents, CDK, or in canister tests like the one for Internet Identity, were the usefulness is quite clear.)

Pinging @kpeacock as the dfx representative here.

5 Likes

With the additional functionality you outlined this sounds pretty useful for developers, I’d be happy to have this!

1 Like

Additional functionality requires serious investment of dev resources, which I don’t see at the moment. And I wouldn’t want to drag this along when it isn’t useful now and won’t be useful soonish.

From our side of things, ic-ref is currently how we do our end-to-end testing in CI for DFX, so it is currently useful in that regard.

We are still discussing what the implications and design goals of building an emulator from the ground up would be, but it is a mid-term goal for the SDK team, and we are hiring for someone who could spearhead that project!

Yes, and I absolutely want to support that! we are using ic-ref-run in Motoko too, and I hope the replica is still using ic-ref-test in their CI.

But that is a controlled environment that does not require us to get ic-ref into a generally distributed, self-contained shape (e.g. static linking etc.) and it does not require us to ship it with dfx.

I agree that from-the-ground-up-emulator might be the better direction.

But that does sound like we could stop distributing ic-ref with dfx and nobody would really notice.

I’ll leave this thread open and see if people have strong feelings, but shipping ic-ref with dfx isn’t critical from my perspective

Ok; I’ll maybe leave this open for another week and if nobody complains I will stop shipping ic-ref and stop building the distribution builds (because then I don’t have to debug the MacOS failures in Bump nixpkgs to 21.11 by nomeata · Pull Request #64 · dfinity/ic-hs · GitHub)

Well, we do still use ic-ref in a MacOS environment for CI - sdk/e2e.yml at master · dfinity/sdk · GitHub. Would stopping support for it become a blocker for us running MacOS e2e tests on CI?

Oh, I somehow assumed you’d build ic-ref from its repo in your CI, not use the dfx-bundled one. Hmm, guess I’ll have to keep it working them, I don’t want to cause regressions in the CI.

Please don’t remove it. For me dfx start --emulator the only way to start local replica on windows wsl2, because on dfx start I get errors:

1 Like