We want to give people an update on the project to make it easier for people to contribute to the IC’s replica codebase.
1. Summary
The Rust code for the replica software as well as core canisters (e.g. NNS) that makes up the Internet Computer has been open and visible since May 2021. But the IC is not updated via pull requests, but rather through NNS proposals which the community votes on via the Network Nervous System (NNS). This all works as intended. The part where design intent and implementation do not match the decentralization needs is that (currently) the code attached to NNS proposals is not visible until after an NNS proposal is voted and executed. This has two practical implications:
Proposals with binary upgrades rely on prose rather than code auditing (not healthy for a decentralized, highly secure project that demands transparency). A recent example: Blessing/Electing Replica Binary (NNS proposal #26833)
Discourages participation from the community.
This is why the DFINITY foundation is addressing the technical hurdles.
2. Technical Background
It is worth explaining why this has not been a trivial effort:
As we want the public to be able to verify that a build that is proposed for an update is created from a specific commit of the repository, we have to ensure that builds are reproducible. Making the code reproducible has required time and effort to identify and eliminate sources of non-determinism in our build process.
Enabling open source contributions in a meaningful way is tricky on an infrastructure level. We have dozens of expensive top-line machines that contribute to the build farm and testnets. These resources are used in CI to gate merges. Furthermore, our complete end-to-end test suite to qualify a full release can take almost a day to run. Our infrastructure and CI system was not originally designed to be exposed to the load and security requirements to process community contributions.
3. Phases and milestones
There is a team working on the following priorities:
Priority 1: Make code publicly available when we vote on proposals; and reproducible so that the community can verify the code matches the proposal.
Priority 2: Show to the public how active and how many people work on the code
Priority 3: Encourage the public to contribute code
Phases
Phase 0: what we have today – squash merge after NNS update. ETA: November 2021
Phase 1 Open source master (possibly also branches) continuously (instead of in fixed intervals as currently). ETA: November 2021
Phase 2: Fully work in the open and encourage community contributions to go through regular CI. ETA: 2022.
If CI really is the main blocker to accept external contributions, it may be possible for a while to do what the Motoko team did until we had a public CI infrastructure: external contributions are initially not CI checked, and after successful review by an internal developer, that developer re-opens a PR with the these changes, which can now go the usual way.
But I am getting ahead of things; transparency into the ongoing development of the replica (and hopefully the Interface Spec as well) would already go a long way.
CI and verify that a build made from the blessed commit creates the same binary as we use for the update (verified builds). From my understanding, making sure the hashes/commits/build numbers all match has been a bit of a pain (because of earlier CI and developer infrastructure decisions), but I will let folks more knowledgeable correct me if I used the wrong terminology.
Thanks, this is actually really important, and I’m glad the foundation takes this seriously.
Is the subnet-by-subnet blessed binary rollout done manually? For example, an internal engineer creates a NNS proposal to update one subnet, waits until it passes, creates another proposal to update another subnet, waits until it passes, etc.
I wonder why that can’t be automated. Can’t some “rollout orchestrator” canister manage the rollout instead?
Priority 3: Encourage the public to contribute code
I’d be really curious what ideas the foundation has on this.
Sorry if I wasn’t clear. I guess there two independent goals in this thread about open sourcing.
Public auditability: Requires providing public read access to the to-be-depoyed code, possibly with history, before NNS deployment, and a way to reproduce the build.
Public contributions: Requires public access to the latest development branch, and a way to submit patches. This benefits from read access to branches, ongoing merge requests, spec documents (ic-ref) and the infrastructure ability to run full CI on such contributions … but it is possible before these are in place. Does not require reproducible builds.
It seems like Phase 1 is pretty much done. I can now trace most of the replica binary update proposals to specific commits in the IC github repo.
I’m curious when Phase 2 will be ready, i.e. open up PRs and scale up the CI.
I think it could be a big multiplier on development velocity, as community members can then submit changes to IC code to implement their motion proposals, instead of asking and waiting for DFINITY engineers to do so on their behalf. Also, it’d be pretty cool to work on the IC codebase.