[Important] Blessing a Replica Binary - Understanding, Promotion, and Voting

[diegop]

“Blessing Replica Binaries”: The friend you did not know you had

Ahoy ICP folks,

I want to talk to you about an important next step in the governance of the Internet Computer: The active promotion of “Blessing replica binary” proposals.

I will first describe the world as it currently is and then discusses the next steps.

What is a “replica binary”? and what does it mean to “bless it”?

A “replica binary” is literally what it sounds: the binary that runs in every node of the Internet Computer. Most colloquially, when folks say that “there is a new update to the IC” what they mean is “there is a new replica binary that is running on every node of the IC.”

Under the hood, it is the binary that is derived from the Rust code here: IC Replica code

These replica binaries typically have lots of updates and features as one would expect from any software development branch ready to hit production.

To “bless the binary” means that there is an NNS proposal for neuron holders to vote that the binary is the one that should be the next version of the IC. Blessing is not enough to have a full update. There are then subsequent NNS proposals to update each subnet with the latest blessed replica binary.

In short, every week or so, there are lots of code updates to fix bugs or add features. They are all tested and rolled into a binary which is voted on.

The Ask: What I am asking the ICP community

We are asking the community to actively look at proposals to bless new replica versions. Have a look look at the release notes, vote, decide to follow the foundation, ask questions, or point out something. We aim for a weekly release cycle. For every new release that we propose to bless we are going to create a new thread.

Here is this week’s release (which is analogous to a “release candidate” in some places):

Bless Replica Binary proposall

Next steps

If a binary is adopted, the following then happens:

1. NNS proposals to update subnets to the newly blessed version are published and voted on.
2. The code in IC Replica code is updated (as a downstream consequence)

Why we are creating this post

We want to encourage more participation in voting in the direction of the Internet Computer as we continue improving transparency.

Background: How long has it worked like this?

Pretty much from the beginning. It has always been there, though the transparency has clearly improved as the dashboards improved, but we want to shine a spotlight on it:

Past replica binary proposals:

• https://dashboard.internetcomputer.org/governance?topic=TOPIC_SUBNET_MANAGEMENT

Future iterations in the process

There are a few areas we want to hang a lantern on and point them out because we are aware they are not great but aim to improve as part of this constant iterative process:

1. Length of voting - these proposals have 48 hours before they expire. In the past, the DFINITY foundation has deliberately not voted in Motion Proposals until the very end to let early voters get their opinions in and the foundation can take it into consideration. Our current thinking for these binary blessing proposals is that the DFINITY foundation (and by extension its followers) will abstain for 12 hours before voting. Could this be 6 hours? 24 hours? Yes, of course. We believe these binaries do not require the large ceremony and deep thinking that designs in motion proposals do. We of course can iterate if find otherwise. Right now it is 12 hours as a starting number to gauge the balance between “moving fast” and "letting people express their votes." Of course, if people do not like how the foundation works, they can stop following the foundation. I expect we will all learn as a community as patterns emerge.

2. Viewing of the code - Right now, the binaries are just that (binaries). The code is visible after it is voted on. This is not ideal so stay tuned for updates on our plan to address this (under the hood: we have a plan we are prepping for the community). Soon, people will be able to see the code, not just the binary, but this is the next step in the current process (giving this NNS proposal more attention).

3. Feature promotion - We will continue to improve the release notes and how features within binaries are verbally promoted so people know when their awaited feature is out. For example, the feature to increase canister smart contracts from 4 GB to 8 GB recently went out in a binary.

4. Making features and NNS proposals tighter - right now there is so much progress to the IC, that a binary has many different features you vote on “all or nothing.” In the future, we have ideas on how we can have more granular voting per feature (using feature flags would be a potential start, for example).

[nomeata]

Good intentions, but I’d say that before the community can

• see the code before voting
• including a useful history (one commit per change, with a useful commit message), so that understanding and reviewing change because feasible,
• and the ability to verify that the binary was indeed the result of the given code

this is just “transparency theater”.

Voting on code changes to the main branch separately and before deployment might also be a way format to make voting more relevant.

Unfortunately, even then this will not quite achieve what we want it to until there is enough expertise outside dfinity and in the community to actually understand and assess the code changes. That will take time (and incentives) … not sure how to do that best.

But maybe this “transparency theater” is still a good way towards real power sharing with the community.

[nomeata]

It is well known that I am personally a big fan of re-inventing the wheel, but sometimes it may be helpful to see how others are solving this problem. So what about the other blockchains that have had on-chain governance, including voting on changes to the protocol and the running binary itself, for longer than we do? Tezos comes to mind…

If anyone here has experience with their approach, I’d love to hear about it!

[skilesare]

@diegop

Bless Replica Binary proposall

Someone was working on a github on the IC. I propose that someone write a compiler in motoko(and all the libraries that it would require) that pulls the code from the IC, compiles the binary, and does it all inside the crypto secure environment of the IC.

I’m only kind of being evil here. I fully expect the IC to do this within some non-absurd timeline.

[diegop]

@nomeata

this is just “transparency theater”.

I think this is a bit harsh, Joachim. (The “theater” part connotes an intent too stop only at the theatre. While we clearly intend to let people know “this is step of 3 out of 50” much more akin to an iterative progress)

(slight simplification of the) options on the table are:

1. Be transparent about an existing process (binaries have been blessed) and make sure community understands, while other parts are set up. Be transparent about the plans.

2. Wait for everything to be perfect to satisfy expectations.

I think quite frankly #2 is done too often, to be honest. I think the IC would benefit from more transparent “this is in flux” culture vs “let me be silent until this is bulletproof” mentality.

I read the intent of your words to mean, “nice first step. Journey not over..” If so, then I think we all agree.

Does that make sense?

(PS: for readers, I have known Joachim for years and have great esteem for him and his work. It is often we are this direct at each other )

[diegop]

@nomeata

Unfortunately, even then this will not quite achieve what we want it to until there is enough expertise outside dfinity and in the community to actually understand and assess the code changes. That will take time (and incentives) … not sure how to do that best.

I agree. To be honest, I have treated this as a secondary problem in my mind to “just getting all the code and docs out there.” But code without the expertise is clearly not enough.

[nomeata]

Yes, maybe a bit harsh … and of course you are right that as a step of many, it goes in the right direction.

I still think there is a bit of theater happening when the community is invited to vote before the vote is meaningful: i.e. there are multiple viable options, and the de-facto power is no longer fully with the foundation and their employees, both in terms of voting power and in terms of expertise.

I am actually at a loss to propose how to get there, short of somehow funding multiple independent entities that collaborate on developer of the IC. Is there a plan for that?

But without that, the voting by the community is … practice? staging for what will be there in the future? But both practice and staging sound like theater

That makes me wonder: what could the community at a large meaningfully vote on already (assuming the staked neurons are already distributed widely enough), given the power and knowledge imbalance? They can veto feature proposals (these votes happen already, so that’s great), and possibly prioritize and help shape the roadmap. So maybe focus on these more meaningful votes first?

Hmm, I’m rambling. Maybe the tl;dr is: I suggest to not pretend that voting on binaries is very meaningful until externals can understand and assess the code changes and reproduce the build, else someone could rightfully call this out as more show than actual power sharing.

[diegop]

@nomeata

They can veto feature proposals (these votes happen already, so that’s great), and possibly prioritize and help shape the roadmap. So maybe focus on these more meaningful votes first?

Well, those are happening…perhaps more concretely, there already are votes on blessing binaries. I do not think the community appreciated that (I think this thread proves that). We want to help educate folks and explain to people the existing process (while improvements occur) rather than let ignorance be bliss.

I suspect there may be two things going on:

1. Perhaps my original writing landed on your ears too much like

"I want to give you a cake! I don’t have eggs, and I haven’t bought all the ingredients yet, but that’s ok. What matters are my cake-giving feelings."

While my intent was more akin to:

“Look, I know we all want cakes. It would be silly of me to keep ignoring that we all want cake. Yes, every week I go buy butter. I do not buy eggs. Those are coming soon”

(If so, as the writer, it is my responsibility to own how things land. It only matters how they land in reader’s minds)

2. I (personally) believe strongly we the foundation and community need to have more of a shared language around imperfection + iteration (a common step for all communities). As we all know, there are many imperfections in any project. I think we need to all be better at " Yes, X is broken. We all see the same thing you do. We have a plan " rather than stay silent at the fear of being criticized.

[nomeata]

I like the cake! Good analogy

[northman]

This sound like you are bringing the ITSM Release process and the positive aspects of a Configuration Control Board process through NNS governance which is a fantastic.

Lke most CCBs in the Enterprise, not everyone will want to rip into the GZIP’d rust code and do a walkthrough. It is there if someone feels so inclined.

Giving a thumbs up to a proposal for me would be: Do the functional changes make sense, has the code been qualified through the defined QA process, is the code baselined per the Configuration Management plan, is there a roll-back plan, are there security critical impacts, has an assessment been completed, is there a rollback plan, is the deployment plan incremental, what are the measurements of success, measure and publish the success. Normal ITSM release management practice.

I agree that this is a transparent approach but I think you will not hear a lot because of lack of expertise and resources outside of the Foundation or Association.

Minor comment, not keen on the term “Blessing” a Replica Binary and prefer using IT Service Management terminology or someething that aligns with CD/CI DevOps wording.

[Manu]

@nomeata

Good intentions, but I’d say that before the community can

• see the code before voting
• including a useful history (one commit per change, with a useful commit message), so that understanding and reviewing change because feasible,
• and the ability to verify that the binary was indeed the result of the given code

this is just “transparency theater”.

I think that’s fair @nomeata. In addition to what @diegop already said, i just want to be super clear that all those three points are being worked on, and I think it’s realistic that we get there within a couple of months.

[jzxchiang]

I feel like it’d be nice to have some on-chain Github for IC code, such that PRs are actually NNS proposals and NNS proposals are actually PRs.

When a NNS proposal passes, then its associated PR is merged into the baseline, which automatically triggers a (hermetic) build of the new source and then deploys the newly created binary onto the replicas, all done on-chain and transparently.

This way, NNS proposals are human-readable (or at least developer-readable), instead of the current situation of working backwards from an opaque binary to determine its source code provenance.

[diegop]

An update on this binary proposal I posted above:

1. The proposal I originally posted is 22701. This had version 3eaf8541c389badbd6cd50fff31e158505f4487d

2. The last blessed revision 3eaf8541c389badbd6cd50fff31e158505f4487d did NOT get deployed to application subnets due to a bug we caught.

3. Yesterday Tuesday, October 5th, the foundation created a new proposal 23633 to bless/elect a new binary which includes the previous work plus the bug fix

https://dashboard.internetcomputer.org/proposal/23633

4. At this point, I should have immediately let the community know and made sure the community had a chance to see before Foundation voted, but I missed the original message so the process failed (and Foundation went ahead and voted with its followers). Therefore proposal 23633 passed to be elected. This gap is entirely my fault as I dropped the ball.

5. The binary, once blessed/elected, started being rolled out as NNS proposal for other subnets (as is the release process):

a. Internet Computer Network Status
b. Internet Computer Network Status
c. Internet Computer Network Status

Update on future process

1. I will keep a closer eye and empower others in foundation and other to communicate these things so I am not the blocker

2. Foundation will soon communicate a plan to achieve the decentralization improvements we all want:

a. Code publicly available when community votes on proposals
b. Enable the public to contribute code

There are some technical hurdles so we will explain the roadmap to get these.

3. Probably not surprisingly, the current intent is to focus as much as we can on #1 (address the technical hurdles) and then iterate on human and community initiatives to encourage more independent parties to contribute.