First off, we want to thank the community for actively participating by thoroughly verifying the proposals and attempting to reproduce the builds. This helps improve the quality of the proposals and supports the decentralization of ICP.
In this case, the DFINITY Foundation will vote to adopt these proposals:
The proposals are for updating guestOS only which has been widely reproduced by the community, as shown in the screenshots in this thread. This means that all code that is being elected by the proposals is fully reproducible.
The releases contains changes that will improve how well the replica can handle the high load we’re currently seeing on mainnet, so we believe adopting these proposals will benefit the ICP community.
Since setupOS and hostOS are not reproducible, we will remove the link to download them from the release page on the dashboard.
The build pipelines include tests to verify that the images are byte for byte reproducible; however they don’t cover a wide enough range of hardware combinations. When you are unable to reproduce a build, sharing the offending artifact can help diagnose and fix the source of indeterminism, and we encourage you to do so proactively. Thanks to everyone who has already shared their images.
In the future we will consider updating the proposal instructions to reproduce builds only for the components that are being impacted - in this case guestOS - in order to avoid confusion.
Edited to reflect both proposals 133309 and 133310
Thanks @Lorimer - The proposals to upgrade hostOS and guestOS are separate and only affect their respective OS. However it is true that the release page for guestOS releases will have a link to the setupOS built from the same commit.
To avoid downloading a non deterministic image of setupOS we will not link it in the release page on the dashboard for these proposals.
Thanks @raymondk, DFINITY’s decision to adopt the release makes sense. Thanks for the heads up.
I don’t think there was confusion in this respect. Note that if only the GuestOS is being built, then only GuestOS code can be reviewed (which goes against discussions from last week). SetupOS/HostOS commits should not be included in the change log if this is to be the case. There needs to be consistency and clarity in this respect.
Nonetheless, I’m glad that the scheduler enhancement will be released sooner, and I’m eager to see the positive effect this will have in production.
Great effort all at DFINITY who have been working on this. It’s a shame this hasn’t panned out quite as expected yet. Any ideas why the actual effect of this change wasn’t picked up during testing?
The incoming hotfix release will address issues found during the rollout of the base release this week, impacting subnet fuqsr. A more extensive analysis of the issues will be posted shortly. It is important to clarify that the issue did not arise due to the HostOS nondeterminism discussed in this thread.
Sorry for this inconvenience with our automation spamming. The bug was found and addressed and will be deployed tomorrow which will resolve this issue.
Please note that some commits may be excluded from this release if they’re not relevant, or not modifying the GuestOS image. Additionally, descriptions of some changes might have been slightly modified to fit the release notes format.
To see a full list of commits added since last release, compare the revisions on GitHub.
Other changes:
f0c923eba Execution,Interface: Revert “feat: Charge canisters for full execution (#1782)”
dfc32fd17 Execution,Interface: Revert “feat: Charge idle canisters for full execution (#1806)”
837588c5e Execution,Interface,Networking: Revert “fix: Tweak instruction overhead per canister (#1819)”
The two SHA256 sums printed above from a) the downloaded CDN image of the GuestOS and b) the locally built GuestOS image, must be identical, and must match the SHA256 from the payload of the NNS proposal. The verification process will also attempt to reproduce HostOS and SetupOS – these results are for advisory purposes only, as this proposal only elects a GuestOS.
Please note that some commits may be excluded from this release if they’re not relevant, or not modifying the GuestOS image. Additionally, descriptions of some changes might have been slightly modified to fit the release notes format.
To see a full list of commits added since last release, compare the revisions on GitHub.
Other changes:
f0c923eba Execution,Interface: Revert “feat: Charge canisters for full execution (#1782)”
dfc32fd17 Execution,Interface: Revert “feat: Charge idle canisters for full execution (#1806)”
837588c5e Execution,Interface,Networking: Revert “fix: Tweak instruction overhead per canister (#1819)”
The two SHA256 sums printed above from a) the downloaded CDN image and b) the locally built image, must be identical, and must match the SHA256 from the payload of the NNS proposal.
While not required for this NNS proposal, as we are only electing a new GuestOS version here, you have the option to verify the build reproducibility of the HostOS by passing --hostos to the script above instead of --guestos, or the SetupOS by passing --setupos.
Doing extensive performance testing with thousands of active canisters is quite slow, so we don’t do that for every single change. In this case, we did not anticipate significant performance changes from scheduler changes. But it turned out that rotating more quickly through all active canisters leads to memory contention which we did not expect.
Thanks Manu. It sounds like an acceptance testing environment would be really useful (aside from the dev environment used for limited testing).
I remember earlier in the year when discussing an incident there was talk of potentially introducing canary nodes that would sort of act as acceptance environments. Did those plans/discussions go anywhere?
Voted to adopt. Build successful, GuestOS hash matches (CDN and local build). HostOS and SetupOS don't, but that's a known issue and they're not the target of this build
There are 3 commits since the previous release. There are 5 files that have been modified. These changes are all reversions of commits from the previous release.
All of these reversions are consistent with the issue that’s been described. This behaves well in production.
This behaves well in production.
