Furthering open source contributons to the IC

Hi folks,

We want to give people an update on the project to make it easier for people to contribute to the IC’s replica codebase.

1. Summary

The Rust code for the replica software as well as core canisters (e.g. NNS) that makes up the Internet Computer has been open and visible since May 2021. But the IC is not updated via pull requests, but rather through NNS proposals which the community votes on via the Network Nervous System (NNS). This all works as intended. The part where design intent and implementation do not match the decentralization needs is that (currently) the code attached to NNS proposals is not visible until after an NNS proposal is voted and executed. This has two practical implications:

  1. Proposals with binary upgrades rely on prose rather than code auditing (not healthy for a decentralized, highly secure project that demands transparency). A recent example: Blessing/Electing Replica Binary (NNS proposal #26833)

  2. Discourages participation from the community.

This is why the DFINITY foundation is addressing the technical hurdles.

2. Technical Background

It is worth explaining why this has not been a trivial effort:

As we want the public to be able to verify that a build that is proposed for an update is created from a specific commit of the repository, we have to ensure that builds are reproducible. Making the code reproducible has required time and effort to identify and eliminate sources of non-determinism in our build process.

Enabling open source contributions in a meaningful way is tricky on an infrastructure level. We have dozens of expensive top-line machines that contribute to the build farm and testnets. These resources are used in CI to gate merges. Furthermore, our complete end-to-end test suite to qualify a full release can take almost a day to run. Our infrastructure and CI system was not originally designed to be exposed to the load and security requirements to process community contributions.

3. Phases and milestones

There is a team working on the following priorities:

  1. Priority 1: Make code publicly available when we vote on proposals; and reproducible so that the community can verify the code matches the proposal.

  2. Priority 2: Show to the public how active and how many people work on the code

  3. Priority 3: Encourage the public to contribute code

Phases

  • Phase 0: what we have today – squash merge after NNS update. ETA: November 2021

  • Phase 1 Open source master (possibly also branches) continuously (instead of in fixed intervals as currently). ETA: November 2021

  • Phase 2: Fully work in the open and encourage community contributions to go through regular CI. ETA: 2022.

14 Likes

Very promising news, thanks!

If CI really is the main blocker to accept external contributions, it may be possible for a while to do what the Motoko team did until we had a public CI infrastructure: external contributions are initially not CI checked, and after successful review by an internal developer, that developer re-opens a PR with the these changes, which can now go the usual way.

But I am getting ahead of things; transparency into the ongoing development of the replica (and hopefully the Interface Spec as well) would already go a long way.

7 Likes

CI and verify that a build made from the blessed commit creates the same binary as we use for the update (verified builds). From my understanding, making sure the hashes/commits/build numbers all match has been a bit of a pain (because of earlier CI and developer infrastructure decisions), but I will let folks more knowledgeable correct me if I used the wrong terminology.

1 Like

Thanks, this is actually really important, and I’m glad the foundation takes this seriously.

Is the subnet-by-subnet blessed binary rollout done manually? For example, an internal engineer creates a NNS proposal to update one subnet, waits until it passes, creates another proposal to update another subnet, waits until it passes, etc.

I wonder why that can’t be automated. Can’t some “rollout orchestrator” canister manage the rollout instead?

Priority 3: Encourage the public to contribute code

I’d be really curious what ideas the foundation has on this.

1 Like

So excited to see this happening!

1 Like

Sorry if I wasn’t clear. I guess there two independent goals in this thread about open sourcing.

  • Public auditability: Requires providing public read access to the to-be-depoyed code, possibly with history, before NNS deployment, and a way to reproduce the build.
  • Public contributions: Requires public access to the latest development branch, and a way to submit patches. This benefits from read access to branches, ongoing merge requests, spec documents (ic-ref) and the infrastructure ability to run full CI on such contributions … but it is possible before these are in place. Does not require reproducible builds.
2 Likes

That is correct. Those are the two goals.

Relevant update: An Update on Furthering open source contributions to the IC

1 Like