Proposal to elect new release rc--2024-06-26_23-01

Hello there!

We are happy to announce that voting is now open for a new IC release.
The NNS proposal is here: IC NNS Proposal 130818.

Here is a summary of the changes since the last release:

Release Notes for release-2024-06-26_23-01-base (2e269c77aa2f6b2353ddad6a4ac3d5ddcac196b1)

Changelog since git revision e3fca54d11e19dc7134e374d9f472c5929f755f9

Features:

  • 787e30c23 Consensus: enable caching in API BNs
  • ae5145d9d Consensus(recovery): Add replay-until-height parameter to ic-recovery and pass it through to ic-replay
  • 1cce4fe68 Consensus(schnorr): Switch ic-recovery to new chain key payload in the update recovery CUP proposal
  • b5e43c1e7 Execution: Add snapshot_id to ChangeDetails::CanisterLoadSnapshot
  • 6a919d072 Execution,Runtime,Networking(execution): Transmit message ID and height for finalised ingress messages.
  • 80c2f8ec0 Networking,Interface: change the wire format for the quic transport
  • 298080696 NNS,Consensus,IDX(ecdsa): Allow adding new keys to an ECDSA subnet

Bugfixes:

  • 2e269c77a Execution,Runtime: Properly handle updating of reserved cycles limit
  • f18cb6e3a Interface: protobuf definitions
  • 19600a436 Message Routing,Utopia: fix StateMachineTests to execute a round in replace_canister_state
  • a0ab22537 Node: Fix chrony driftfile
  • dd0bae015 Node: Allow systemd to relabel journal files
  • df93219d9 Node: Allow node_exporter to get attributes in /proc
  • 3ca6b762e Node: update node_exporter option after version update
  • b61565fae Node: Fix parsing of deployment.json

Chores:

  • cd28507fc Boundary Nodes,Node: add configuration options
  • 0360f812c Consensus: donā€™t pass in the membership struct for certification and https outcalls
  • 3c973a316 Consensus(schnorr): Set deprecated EcdsaKeyId in key transcript to None
  • 639dccf86 Consensus,Boundary Nodes,NNS: fixing a few typos
  • 65e359daa Crypto: rename ic-crypto-tecdsa to ic-crypto-utils-canister-threshold-sig
  • 042793811 Crypto: make conditionally_add_delegation_cert reproducible
  • 40eca0103 Execution,Runtime: Check for Wasm benchmarks dependencies
  • e5ed52067 Interface: (re)move and upgrade deps
  • daa0f9b5d Interface: merge context_logger with the logger crate
  • 1a5c233f3 Node: Update container base images refs [2024-06-25-2112]
  • 447ea7917 Node: Update container base images refs [2024-06-20-0850]
  • d8731aa6d Runtime(RUN): Upgrade Wasmtime to 21
  • fdf288f55 Runtime,Interface,NNS: upgrade crates and use workspace versions

Refactoring:

  • af4afd80f Execution,Runtime: switch sign_with_ecdsa to use SignWithThresholdContext
  • d59cdb48c Interface: remove unused protos
  • eee57c7b3 Message Routing: Change push signals API for Stream
  • 19ffb1746 Node: Remove build_container_filesystem_config.txt

Tests:

  • 95075aea7 Consensus,IDX(schnorr): Add Schnorr keys to some existing ECDSA system tests
  • d1e6c8eaa Execution,Runtime: Temporary disable page size related tests to build ic on Apple M* platform
  • e7cf8c8de Execution,Runtime,Message Routing,Utopia: Add current_interval_length into StateMachine lib
  • 8ab41904c Message Routing,Utopia: Stop making checkpoints in tests that donā€™t flush expected_wasm_binaries
  • 2174daef9 Runtime: Make sandbox tests fail instead of hanging on panics

Other changes:

  • 6cedc4a0a revert: chore: update the node_exporter version on the guest and host OS
  • ed5ac23c3 Revert ā€œChore: Update container base images refs [2024-06-20-0850]ā€

IC-OS Verification

To build and verify the IC-OS disk image, run:

# From https://github.com/dfinity/ic#verifying-releases
sudo apt-get install -y curl && curl --proto '=https' --tlsv1.2 -sSLO https://raw.githubusercontent.com/dfinity/ic/2e269c77aa2f6b2353ddad6a4ac3d5ddcac196b1/gitlab-ci/tools/repro-check.sh && chmod +x repro-check.sh && ./repro-check.sh -c 2e269c77aa2f6b2353ddad6a4ac3d5ddcac196b1

The two SHA256 sums printed above from a) the downloaded CDN image and b) the locally built image, must be identical, and must match the SHA256 from the payload of the NNS proposal.

3 Likes

Hello there!

We are happy to announce that voting is now open for a new IC release.
The NNS proposal is here: IC NNS Proposal 130819.

Here is a summary of the changes since the last release:

Release Notes for release-2024-06-26_23-01-storage-layer-disabled (b6c3687fb3a03ca65fcd49f0aadc499367904c8b)

Changelog since git revision 2e269c77aa2f6b2353ddad6a4ac3d5ddcac196b1

Bugfixes:

  • b6c3687fb Interface: Disable new storage layer

IC-OS Verification

To build and verify the IC-OS disk image, run:

# From https://github.com/dfinity/ic#verifying-releases
sudo apt-get install -y curl && curl --proto '=https' --tlsv1.2 -sSLO https://raw.githubusercontent.com/dfinity/ic/b6c3687fb3a03ca65fcd49f0aadc499367904c8b/gitlab-ci/tools/repro-check.sh && chmod +x repro-check.sh && ./repro-check.sh -c b6c3687fb3a03ca65fcd49f0aadc499367904c8b

The two SHA256 sums printed above from a) the downloaded CDN image and b) the locally built image, must be identical, and must match the SHA256 from the payload of the NNS proposal.

3 Likes

Thanks for posting this release @DRE-Team. I have a few questions about the proposal summary if thatā€™s okay:

  • Can I ask why 8e7a89e4 wasnā€™t referenced by the proposal summary (deprecating ECDSA-specific fields)
  • Not all commits that have made changes to ā€˜rs/tests/src/orchestratorā€™, ā€˜rs/tests/src/tecdsaā€™, ā€˜rs/tests/src/consensusā€™ and ā€˜rs/tests/src/driver/ā€™ files are included in this proposal summary. Iā€™m guessing that lack of an entry in the CODEOWNERS lookup file for ā€˜rs/tests/src/driver/ā€™ is the reason for one of these omissions (first point above). But Iā€™m not sure about the others. Are you able to clarify?
  • Interestingly, a change to the registry canister has been picked up as a relevant change and referenced in this proposal summary (but as far as I understand, it doesnā€™t actually modify GuestOS). Accepting this proposal wont actually deploy this change. Is it right to include this change in the proposal summary? (I suspect itā€™s due to the proposal summary tooling picking up on the change to the test file, but still)
  • This proposal summary also references a commit which is then later reverted by another commit. Is there any chance of excluding (or marking) commits that have been reverted within the same proposal in future summaries (to avoid confusion)?

Thanks in advance. I always find your responses very helpful :pray:

2 Likes

As a separate question, I noticed that node exporter is being downgraded to an earlier version, but thereā€™s no mention of why (that Iā€™ve been able to find). The commit references an internal GitLab MR. Can I ask how far away DFINITY are from making this sort of information public? @basvandijk :pray:

1 Like

Hopefully this thread is the correct place for discussing the HostOS election proposal that accompanies these GuestOS proposals.

Iā€™m left feeling a little confused by the HostOS proposal summary. Iā€™m hoping to treat this as a learning opportunity. If I had to wrap up my confusion into a simple question, itā€™s - which of the commits referenced in the HostOS proposal summary actually modify the behaviour of HostOS, and why are other commits included in the proposal summary? Is the adjustment to boundary node firewall configuration actually considered a HostOS change? What about the change to allow journal file relabelling?

Which of the changes actually voted in by this proposal actually take affect via this proposal?

Thanks in advance @DRE-Team

1 Like

Reviewers for the CodeGov project have completed our review of these replica updates.

Proposal ID: 130318
Vote: ADOPT
Full report on OpenChat

Proposal ID: 130319
Vote: ADOPT
Full report on OpenChat

At the time of this comment on the forum there are still 2 days left in the voting period, which means there is still plenty of time for others to review the proposal and vote independently.

We had several very good reviews of the Release Notes on these proposals by @Zane, @cyberowl, @ZackDS, @massimoalbarello, @ilbert, @hpeebles and @Lorimer. The IC-OS Verification was also performed by @tiago89. I recommend folks take a look and see the excellent work that was performed on these reviews by the entire CodeGov team. Feel free to comment here or in the thread of each respective proposal in our community on OpenChat if you have any questions or suggestions about these reviews.

4 Likes

Hi @Lorimer , thanks for asking the questions HostOS rollouts are still not fully automated since we submit these proposals much less frequently and we only submitted a few of them so far. But weā€™ll get there. In the meantimeā€¦ a few more messages on these forum posts should be fine I think.

To discover which changes affect HostOS, we find all commits that may impact the HostOS update image (since the last release):

In particular the adjustment to boundary node firewall is indeed a component that is shared between the boundary nodes and the HostOS. The journal selinux relabeling very likely takes effect on the HostOS as well ā€“ node team would have to confirm that if you consider this question important?

All changes in the proposal should take effect after the HostOS upgrade. At least if we ask Bazel (build system). And itā€™s hard to argue against the tooling :smiley:

Since I have your attentionā€¦ with CodeGov review the HostOS proposals as well? The release cadence should be approximately once per quarter, although we donā€™t have anything set in stone yet.

3 Likes

Yes, CodeGov will review these HostOS upgrade proposals since they fall under the IC-OS Version Election proposal topic. We are committed to review all proposals that fall under this topic as well as the System Canister Management proposal topic. Itā€™s very helpful to know the cadence of each type of proposal and for it to be consistent, so thank you for giving that consideration.

3 Likes

Thanks for providing more info @sat.

Iā€™ve had the impression that the build system isnā€™t the most reliable judge for what changes are actually modifying the target system for a specific proposal. This came up a few weeks ago when discussing why the proposal summary generation tool needs to filter various things out.

Other tooling limitations donā€™t appear to be uncommon.

The boundary node firewall caught my eye because firewall changes have previously been deployed via GuestOS proposals. Iā€™m curious why some firewall changes take effect via HostOS proposals and others via GuestOS (which runs inside a VM spun up by HostOS). Itā€™s not too important, but if someone from the node team is able to take 5 mins to clarify/explain this, that would be awesome.

The change to the registry canister is another example (mentioned in my first post above). I donā€™t believe that change actually modifies GuestOS behaviour - does it?

Would I be overstepping if I created a new forum topic with an aim to discuss and brainstorm ways of making proposal summaries more reliable and/or easier for the community to follow/understand/validate? I think this sort of conversation spans more proposals than just this weeks ones. Having a consistent place to catalogue proposal accuracy issues and discuss/rule out potential ways to improve things would be useful (I hope other in the community may get involved also - hopefully constructively).

3 Likes

Hey @Lorimer, absolutely no problem to create another forum post to discuss release notes generation. Actually, you would also be more than welcome to create PRs for the release notes generation tooling.

Also, the reason we have tooling for this (instead of doing it by hand) is because tooling gives us much better cost-benefit ratio. I wouldnā€™t be against other options as long as it doesnā€™t cost us a lot more.

Regarding the node questions, let me poke @andrewbattat, he should be able to provide the information you ask for.

4 Likes

Hi Lorimer! GuestOS still dynamically manages its own firewall via NNS proposals. The HostOS firewall is static and much smaller in scope.

Thanks @andrewbattat, but are you able to provide a little more detail? Are you confirming that the correct deployment target for this commit was HostOS or GuestOS?

The nftables config that youā€™ve linked to doesnā€™t appear to have been modified since the repository reorganisation a month ago. The last meaningful modification appears to have been this one back in May. Note that this wasnā€™t referenced in the HostOS proposal, and was instead referenced in this GuestOS proposal (130134).

1 Like

The deployment target for this commit was GuestOS. To see the full list of files copied from the components/ folder to each ic-os variant, open each variantā€™s respective .bzl file.

But note that this commit is not a firewall change. Instead, this change whitelists an additional IPv6 prefix for tests that depend on an SSH connection to a development GuestOS imageā€”this change will have no practical effect on production GuestOS images as SSH is disabled. No changes have actually been made to the firewall itself. So youā€™re correct that the nftables config that I linked to is rarely modified, as it should be :+1:

You are getting at something important, and something that Iā€™ve pushed for internally, which is better organization of our proposal releases. Many changes to the HostOS are included in GuestOS proposals (and same for changes to SetupOS). This is because we release GuestOS proposals every week, so it defaults as our release notes for the whole IC-OS. Then, every few months when we do HostOS releases, the HostOS proposal only includes ic-os changes from the last week or so (and it includes GuestOS changes, as weā€™ve just seen). This system is imperfect.

My personal vision is to have a weekly ā€œIC-OSā€ release that would include changes for all IC-OS images, and then after the release is approved by the community, we would do GuestOS/HostOS upgrades based on the latest approved IC-OS release. This is more of DREā€™s domain though (@sat) :slight_smile:

1 Like