We have identified a security issue and are following the procedure described in NNS proposal 48792: Security Patch Policy and Procedure. An NNS proposal containing the fix will be submitted shortly. More information about what is being released will be posted here after the rollout of the security hotfix completed.
4 Likes
Thanks @derlerd-dfinity1. I’ll be interested to learn what the issue was, and why this disproportionately affected the lhg73 subnet (again). It’s one of the largest subnets, particulalry in terms of memory usage, but it’s not the largest.
I’m not sure what makes the subnet special/susceptible. But I’d like to understand this.
1 Like
We submitted 2 proposals:
https://dashboard.internetcomputer.org/proposal/134608
which builds on top of 3e24396441e4c7380928d4e8b4ccff7de77d0e7e, and
https://dashboard.internetcomputer.org/proposal/134609
which build on top of d9fe2076f677a08734bed90c67b1c3f4056ed621
We plan to update each subnet to its current version + the fix above.
Here is a list of all subnets and their current versions:
4 Likes
The explanation will sadly have to wait until we’ve finished the rollout process. But it won’t take long to complete that. We don’t estimate the change/fix to be risky, so we should be able to quickly roll it out.
4 Likes
With security patch policy, public can’t verify what was released. How long will this policy stay ?
1 Like
Can you clarify the question @aligatorr89 ? Do you mean how long will the policy stay in this case, or in general?
The policy was adopted as a motion proposal a while ago. We can of course do something else instead, but the no1 requirement is that it’s safe for the IC (e.g. can’t be exploited by malicious parties). Please make suggestions if you have any - what should be done in case there are security-sensitive fixes that need to be rolled out?
4 Likes
This policy was adopted shortly after genesis (probably in late 2021). I doubt this policy will change any time soon nor would I want it to change any time soon. We are talking about emergency response to security risks and the community is very far from being a part of the solution.
2 Likes
Yes, if policy for security patches will be every eased. Since knowing the vulnerability can cause bad actors to exploit, it’s better to stay secret, but in the long run fixes have to become transperant from the begging.
For the solution, only one I can think of is code being unexploitable. So these security patches don’t happen anymore.
1 Like
The security hotfix has meanwhile reached all subnets. It was necessary because of an issue in the hashes in blocks feature. As was seen, the issue can lead to some nodes in a subnet wrongly assembling blocks in a way that looks as if a valid block is invalid. When enough nodes assemble the blocks like that, Consensus can no longer make progress and the subnet gets stuck. While we are woking on properly fixing the issue, the hashes in blocks features is switched off in the recent security hotfix. The respective branches (here, and here) have meanwhile been pushed to the IC repository and the community can now verify the builds.
We will follow up with a more detailed post mortem as soon as possible.
7 Likes
Thanks @derlerd-dfinity1. Are you able to comment at this stage on why the lhg73 subnet has been disproportionately affected? (and the prior stall several months back)
1 Like
Please note that I cast a manual vote with the CodeGov neuron today on IC-OS Version Election proposals 134608 and 134609. Since these proposal were part of this security hotfix, they had already been voted by DFINITY. I voted about 52 hours after the proposals were submitted and after @ZackDS and @hpeebles has completed their reviews and voting. Hence, it was an informed vote based on the information available at the time and after the 48 hour deadline that the CodeGov team has historically agreed to meet. I cast the votes manually because Hamish and I were doing some troubleshooting on the WaterNeuron vote relay app and I need to make sure we had some examples to look at. The rest of the team is still planning to complete their reviews, but I wanted to make sure folks know that our CodeGov vote was cast prior to reaching consensus of the Followees of our neuron.
4 Likes
Proposal 134608
Summary
- Vote: Adopt
- Hash: Hashes match
- Reasons to adopt: Builds fine + hashes match
Proposal 134609
Summary
- Vote: Adopt
- Hash: Hashes match
- Reasons to adopt: Builds fine + hashes match
1 Like
@Lorimer I think it would be best to wait with details until the post mortem is ready if you don’t mind.
2 Likes
Proposals 134608 & 134609 | Tim - CodeGov
Vote: Adopt
Reason: Critical hotfixes as explained in the proposal texts, already executed. Builds successful and hashes match.
About CodeGov…
CodeGov has a team of developers who review and vote independently on the following proposal topics: IC-OS Version Election, Protocol Canister Management, Subnet Management, Node Admin and Participant Management. The CodeGov NNS known neuron is configured to follow our reviewers on these topics and Synapse on most other topics. We strive to be a credible and reliable Followee option that votes on every proposal and every proposal topic in the NNS. We also support decentralisation of SNS projects such as WaterNeuron and KongSwap with a known neuron and credible Followees.
6 Likes
I’m sorry I wasn’t able to review the proposals in time before the voting period came to an end.
4 Likes
No worries. Is there any update on when the post-mortem is likely to be available?
2 Likes