[Feedback Wanted] Security Patch Policy and Procedure

Developers
1

Introduction

The DFINITY Security team and R&D team are proposing an open policy and procedure for handling high-risk security patches of the IC. The team is looking for feedback, questions, suggestions, etc…

Discussion leads

  • Raghavan Sundaravaradan, Engineering Manager (Security) at DFINITY (@rsundar01)
  • Sam Burri, VP of Engineering at DFINITY (@samuelburri )
  • Ali Piccioni, Team Lead, Internal Developer Experience at DFINITY (@ali.piccioni)

Summary

Here we will be discussing the policy and procedure for handling security patches and how it affects open-sourcing the Internet Computer software. The scope of this policy includes the replica software running on node machines and system canisters in the Network Nervous System (NNS) subnet. Currently, the Master and Release Candidate branches of the IC internal GitLab repositories are mirrored to the public IC GitHub repository. So effectively any commits to the Release candidate branches are visible to the public almost immediately. But there are situations where openness can be in direct conflict with security and the most important of them is when the system is being patched for security vulnerabilities. Here we will document the policy and procedure surrounding the application of security patches

Policy

Openness is a top priority for the DFINITY Foundation. The community’s participation in shaping and governing the Internet Computer is an important driver for its success. This is evident from the series of open-sourcing efforts that took place following the Genesis event and the foundation’s efforts to shape the governance system into an instrument for community participation. All major changes to the Internet Computer, including changes and upgrades to the platform, are subjected to a proposal and voting process that keeps the community updated and enables members to discuss, approve, and reject proposed changes as well as verify them. It is important to be clear here: openness is key to the IC (and all code upgrades will be public), this policy is the DFINITY foundation being open how it best thinks it can address high CVEs (and protect everyone) sometimes depends on exposing the code upgrade after the proposal. Security patches are very sensitive by nature, and knowledge of a patch can provide a window of opportunity for an adversary to mount an attack on the system before it has been patched. As a result, the DFINITY Foundation has adopted the following policies:

  1. Due to the sensitive nature of security patches, the foundation will not make the content of security patches public until the entire system has been patched.
  2. As with any other upgrades, a proposal will be submitted for the community to vote on, aka “bless”, a new version of the artifact. The proposal will contain the checksum of the Internet Computer deployment artifacts. But the main difference is that the community won’t be able to verify the checksum as the code is not yet published.
  3. The details of the patch will be revealed after a quiet period of 10 days from the time after the successful application of patches to the entire system. The patch activity is considered successful if the applied patch completely mitigated the vulnerability as intended and the system remained stable without any negative side effects as a result of the patch. At this point, the community can verify the checksum included in the proposal in step 2.

What is a security patch?

A security patch is an update to software or firmware that mitigates existing known vulnerabilities present in them. For the Internet Computer, the security patches applied to the replica software and system canisters running in the Network Nervous System (NNS) are covered by this policy.

Why is a security patch sensitive?

The release of a security patch before applying it to the intended system may provide adversaries with adequate information on the anatomy of the vulnerability it is trying to protect and how to exploit it for their gain.

What is the need for the quiet period?

The Internet Computer is a complex piece of software with several distributed nodes. Despite comprehensive testing of the security patch, it is possible that something goes wrong. The quiet period provides ample time to ensure the stability of the system, and would allow a rollback if necessary before the content of the patch is revealed.

When does the quiet period start?

Due to the complexity and distributed nature of the Internet Computer, patches and upgrades happen in a staggered manner, covering a few subnets at a time, and the entire process may take several days depending on the complexity of the patch and upgrade. The quiet period starts after 100 percent of the platform has been patched., which is typically when then NNS subnet has been upgraded.

5 Likes
2

We are very much looking for feedback. We are aware that security is extremely important to any crypto project.

We want to formulate a policy we can post on the dfinity.org website so it is clear and obvious to all. We believe this proposal is within the norms of the industry (e.g. how projects behave), but we want to go a bit further and openly describe it so we can receive feedback and iterate.

3

Can you say more about what “Handle Exception” entails?

1 Like
4

Update: I have updated the main section because I had some sections which were NOT part of what would go on the website policy (that was my fault). So I have edited to reflect what is the policy (not implementation details and resources).

5

Hi There, Thanks for the question. The exceptions in the chart speak for extreme corner cases which are unlikely to happen but nevertheless, the possibility exists and it is been explicitly mentioned there. In case of such instances, it needs to be handled on a case-by-case basis. The 2 places in the process where this has been specified is:

  1. The process flow if the proposal wasn’t accepted. If this is the case I would imagine that the foundation will have to engage with the community or revert to the usual upgrade process, etc.
  2. The other situation is where there are no security patches available for exposure. It is hard to imagine a situation where even a compensatory control is not available. Again this needs to be handled appropriately taking into account all the possibilities, risks, and commitments.

Also, one thing to note is that the procedure and chart are more of a direction for internal teams that are handling security patches. So an exception highlights the need for more discussions in handling the case.

2 Likes
6

Proposal is now live: Internet Computer Network Status

1 Like
7

I totally missed this proposal. It’s reassuring to know that DFINITY was paying attention to the Wormhole / Solana hack, and working on a mitigation for the IC.

A couple questions, if it’s not too late:

  1. Will security patches be separated into their own proposals instead of bundled into replica upgrade proposals? If they are bundled together, then the community would not be able to verify the checksum of the bundled proposal, I think.

  2. Is there a human in the loop involved in making the patch source code available? In other words, after the quiet period ends and all subnets have been upgraded, will the patch code automatically be made public, or does a DFINITY engineer need to “press a button”?

  3. This means the public GitHub master branch will diverge from the internal GitLab master branch, at least until after the quiet period for a patch. Will the public master eventually catch up to the internal master and reflect the live IC?

  4. If not the source, what kind of details will you put in the proposal description so the community can make an educated vote? Or will we have to just trust DFINITY on this?

1 Like
8

Hey @jzxchiang! Thanks for the questions!

  1. Yes the proposal for the security patches will be separate. The commit hash and the artifact checksum will be provided as usual, but the code will be kept private until the patch can be applied and the checksum cannot be verified during that time.
  2. At this point in time we are not thinking about any automation, so this will be a manual step
  3. Once the patch has been successfully applied the public GitHub ic repo will be back to mirroring the GitLab master branch.
  4. Please refer to 1.

Please LMK if you have more questions.

2 Likes
9

This looks great!

@diegop Is there a rollback mechanism, such that the community could vote to rollback said piece of software to a previous iteration if all of a sudden there are performance, or other unintended side-effects such as breaking changes?

1 Like
10

@icme I pinged someone from the tea to answer your question. I want to make sure the answer is 100% accurate (apologies for delay).

11

Update:

The website now shows the security policy: Introduction :: Internet Computer

12

@diegop The link you provided looks to be a link to security best practices for developers.

https://smartcontracts.org/docs/security-best-practices/introduction.html

I can’t find information on the patch procedure guidelines at this url.

I cleared my browser cache just to make sure, but don’t see any new updates to the page relevant to this post.

Screen Shot 2022-03-11 at 13.03.52
Screen Shot 2022-03-11 at 13.03.521920×757 95.4 KB

1 Like
13

Oops you are totally right.

I actually have the RIGHT link, but the wrong forum thread. i meant to post this on a forum thread about best practices.

14

@icme great question. You can always create a new NNS proposal to propose that the network go back to the previous version. We do run tests for piecewise upwards and downwards compatibility.

2 Likes