No IC OS release proposal this week


We have decided to not submit a proposal for electing a new IC OS version this week.

Earlier this week, we discovered a non-reproducibility issue (which, fortunately, is now well on its way to being diagnosed and fixed). This issue manifests in the HTTP outcalls canister, whose build output currently results in different checksums from time to time. Since the canister is part of the IC OS release, this cascades into affecting the reproducibility of the IC OS release.

We cannot in good faith ship an IC OS release that cannot be reproduced independently by the IC community. Therefore, we have decided to postpone the release to next week. To ensure that we discover non-reproducibility issues sooner, we’re also improving our testing stack to test reproducibility in more diverse environments.

For transparency purposes, what follows would have been the release notes for this cycle’s release. We’re eager to bring all these improvements to you — and more — in next week’s cycle.

Release Notes for release-2024-03-14_23-01-default (d51f0f58f899e404be9bc0b60a0d35df53cb2553, scrubbed)

Changelog since git revision 778d2bb870f858952ca9fbe69324f9864e3cf5e7


  • b7d910d85 (author: Igor Novg) Boundary Nodes: ic-boundary: add journald logging
  • 73278306d (author: Jack Lloy) Crypto: Support ring’s buggy PKCS8 format for Ed25519 keys
  • 1d9661c50 (author: Alex Zapu) Execution,Runtime: Implement take canister snapshot
  • a1a201bd0 (author: Alex Zapu) Execution,Runtime: EXC:1532: Introduce delete canister snapshot arguments
  • c6ad509b8 (author: Andr Bere) Execution,Runtime: Query Cache: Cache all but the transient errors
  • 608113a89 (author: Math Björ) Financial Integrations(icp_index): Add SettledTransaction with TimeStamp field to ICP index
  • f4e5d56f9 (author: Alin Sinp) Message Routing: New certification version with Request and Response deadlines
  • 57bd87670 (author: Alek Pakh) Message Routing,Runtime: merge sharded overlays
  • 147403cca (author: Dani Shar) Networking(http-endpoint): Add HTTP/2 support to alpn protocol header
  • ba9640a49 (author: Tim Gret) Networking,Consensus(consensus/p2p): push artifacts based on importance v2
  • 6340230d1 (author: Jaso (Yel) NNS: Define an enum DissolveStateAndAge to represent valid and invalid combinations of dissolve state and age
  • ff34c8335 (author: Arsh Ter-) NNS(neurons-fund): Set Neurons’ Fund participation limits based on Network Economics and XDR/ICP conversion rate
  • fbc9b6182 (author: Dani Wong) NNS(sns): From valuation, calculate minting upper bound.
  • 11f35faf2 (author: Arsh Ter-) NNS(neurons-fund): Add [minimum|maximum]_icp_xdr_rate to NeuronsFundEconomics
  • b7faf18d8 (author: Dani Wong) NNS(sns): Calculate total of minting proposals in the past 7 days.
  • bbe7a1864 (author: Arsh Ter-) NNS: Extend NNS Governance state with xdr_conversion_rate
  • a072df420 (author: Gary McEl) Node: Bazel base image target
  • 5482ca3bc (author: Andr Batt) Node: Rename log_and_halt_installation_on_error
  • 626d1fd79 (author: Andr Batt) Node: Remove python vsock code
  • 5d94ab31d (author: Math Björ) Node,Crypto(crypto): Add fstrim_tool to run fstrim and collect metrics
  • 9ec68e222 (author: Maks Arut) Runtime,Execution: Add trap message to canister logging


  • ea37f353e (author: Rüdi Birk) Boundary Nodes,Node(nginx): intercept errors and display custom error page
  • 79de489c4 (author: Rüdi Birk) Boundary Nodes,Node(static-testnet): allow deployments without denylist
  • 8e4dc77fb (author: Andr Bere) Execution,Runtime: Re-classify hypervisor ic0.call_cycles_add trap as ContractViolation
  • b9fdc7d9c (author: Tim Gret) Networking(consensus_manager): disable request size limit
  • cae6cdb8b (author: Andr Batt) Node: ipv4 connectivity endpoints


  • ba7d51c98 (author: Rost Rume) General: remove unused deps
  • c8b982723 (author: Niko Koma) Boundary Nodes(BOUN): increase http connect timeout in ic-boundary
  • 7625fb2c1 (author: Mart Rasz) Boundary Nodes: add DecoderConfig when decoding candid in BN code
  • 541a311fd (author: Andr Batt) Consensus: Improve HostOS console logging during registration
  • d41ccefd9 (author: Kami Popi) Consensus: Use warn! instead of trace! in batch delivery when there is no finalized block at the expected_batch_height
  • cf03c9d5b (author: Jack Lloy) Crypto: Upgrade curve25519-dalek crate to 4.1.1
  • 808d4c01b (author: Rost Rume) Crypto: use the workspace version for x509-parser
  • 635db6088 (author: Jack Lloy) Crypto: Address Ed25519 crate review comments
  • 0a9e2d923 (author: Mart Rasz) Execution: bump candid to 0.10.5 and ic-cdk to 0.13.1
  • 334bc6082 (author: Maks Arut) Execution,Runtime: Always return error for fetching canister logs as ingress despite the feature flag
  • 80dc31b92 (author: Andr Bere) Execution,Runtime: Remove unused InstallCodeContextError::InvalidCanisterId
  • 5fb072c88 (author: Niko Haim) Financial Integrations(ICRC-Rosetta): multiple ingress intervals
  • 0fe5aff1e (author: Dani Shar) Financial Integrations,NNS: Move num-traits dependency to workspace
  • 4495e2b5e (author: Andr Bere) T&V,Interface: Review HypervisorErrors
  • 5eda0c87d (author: Adam Brat) Interface(RUN): Remove unused HypervisorErrors
  • b083a168f (author: Rost Rume) Networking: small cleanup
  • 694fe304c (author: Mart Rasz) NNS: add DecoderConfig when decoding candid in canister code
  • 51e808008 (author: Saša Tomi) NNS(nns,sns): Bump up the NNS/SNS proposal summary limit to 30KB
  • 3a604a703 (author: Andr Batt) Node: Clean up IC-OS configuration documentation
  • 89ed76b01 (author: Andr Batt) Node: Remove old GuestOS documentation
  • 838075e5b (author: Adam Brat) Runtime(RUN): Remove error on bad validation config
  • e3755333d (author: Adam Brat) Runtime(RUN): Remove unused deserialize error


  • 725a52e61 (author: Jack Lloy) Crypto: Remove EccFieldElement abstraction
  • b969becdd (author: Fran Prei) Crypto: rename PreSignatureQuadruple to EcdsaPreSignatureQuadruple
  • 5efe7af00 (author: Mart Köni) IDX(bazel,candid-integration-test): move to separate job
  • 98c225e28 (author: Stef Schn) Message Routing: Move PageMapLayout to state_layout
  • b19657cf5 (author: Rost Rume) Networking: remove the SEV dep from the replica
  • c6894dfa1 (author: Jaso (Yel) NNS: Define a feature flag and differentiate old and new merge neuron flows
  • ea69f6f41 (author: Drag Duri) NNS: Remove stable structures from base_types


  • 838549128 (author: Leo Eich) Consensus(exhaustive): Extend ExhaustiveSet unit test
  • 2525c94f0 (author: Alin Sinp) Consensus: Drop unused ic_test_utilities_consensus code and dependencies
  • a831c94f2 (author: Alin Sinp) Consensus: Split off an ic-test-utilities-consensus crate from ic-test-utilities
  • 56579e061 (author: Grég Dema) Cross Chain(ckerc20): stop ckETH minter before upgrading it in state machine tests
  • 755e49e16 (author: Niko Haim) Financial Integrations(ICRC-Rosetta): test for payloads
  • 283092b57 (author: Leo Eich) IDX(publish): Publish types unit test binary to CDN
  • 44b1cd8db (author: Alek Pakh) Message Routing: Add a test for stability of shard size.
  • d16dec742 (author: Shuo Wang) Message Routing: Refactor state sync tests to test the public interface
  • a29c1907f (author: Alin Sinp) Message Routing,Interface: Move ic_test_utilities::notification into the only test using it
  • aac08d984 (author: Arsh Ter-) NNS: Bump max time expectation for Root metrics to 0.050 s
  • 043082992 (author: Arsh Ter-) NNS: Introduce protobuf builder for NNS Governance testing
  • 3f9655c9f (author: Arsh Ter-) NNS: Fix Pocket-IC helper functions for upgrading NNS Governance
  • 8b4abff88 (author: Alin Sinp) Runtime: Split off an ic-test-utilities-embedders crate from ic-test-utilities


  • 6e045f4e8 (author: Olek Tkac) Crypto: fix comment in tECDSA

Other changes:

  • 2f6164518 (author: Igor Novg) Boundary Nodes,Node: leave slicing only on raw domain
  • f61c56422 (author: Rost Rume) Consensus,Boundary Nodes,Node: don’t suffix the newer clap version
  • 5cd5d6568 (author: Rost Rume) Consensus,Crypto,Interface: yet another attempt to fix the time source complexity we currently have
  • 4372669f5 (author: Alin Sinp) Consensus,Execution,Message Routing: Split off an ic-test-utilities-types crate from ic-test-utilities
  • 98c0060a2 (author: Math Björ) Consensus,Networking,Boundary Nodes,Node,Financial Integrations: don’t suffix the newer clap version (cargo edition)
  • df2b015a4 (author: Alin Sinp) Execution,Consensus,Message Routing,Interface: Split off an ic-test-utilities-state crate from ic-test-utilities
  • e6913a356 (author: Andr Bere) T&V,Execution,Runtime,Interface: Change some error codes to keep the convention
  • 419408154 (author: Mark Kosm) IDX: Revert “Merge branch ‘martin/chore-candid-add-manual-flag’ into ‘master’”
  • 4e9578cb0 (author: Alin Sinp) Message Routing,Execution,Interface: Add deadline fields to messages, callbacks and call contexts
  • 32037614a (author: DFIN GitL) Node: Updating container base images refs [2024-03-14-0814]
  • 310086145 (author: DFIN GitL) Node: Updating container base images refs [2024-03-07-2303]
  • 816c2a0df (author: Eero Kell) Node: Continue to use SEV kernel on SetupOS
  • fc0ba734c (author: DFIN GitL) Node: Updating container base images refs [2024-03-07-0812]

Thanks for sharing, let us know if we can help with anything.

1 Like

I’m happy to report that we are making progress identifying the source of the nonreproducibility. I can’t say for sure when the problem will be solved because I don’t want to risk sounding premature — but we will soon update with more information.


I noticed that the p2p version still got released according to Release release-2024-03-14_23-01-p2p · dfinity/ic · GitHub . Is that just for testing ?

1 Like

Keen eye! It got branched and tagged, but no proposals will be submitted to adopt it.


Hey, any update on this ? Tried to replicate the issue and couldn’t built it 12 times on different specs all built successfully and hashes matched. But then again with the latest release from this Friday managed to build it with no issues and even though nothing changed just a simple re proposal for the explained issues, last night and today I failed 5 times with the HostOS hash mismatch. Rest of the team managed to re build it normally.