Proposal to elect new release rc--2024-10-23_03-07

Hello there!

We are happy to announce that voting is now open for a new IC release.
The NNS proposal is here: IC NNS Proposal 133798.

Here is a summary of the changes since the last release:

Release Notes for release-2024-10-23_03-07-base (75dd48c38f296fc907c269263f96633fa8a29d0e)

This release is based on changes since release-2024-10-17_03-07-base (e54d3fa34ded227c885d04e64505fa4b5d564743).

Please note that some commits may be excluded from this release if they’re not relevant, or not modifying the GuestOS image. Additionally, descriptions of some changes might have been slightly modified to fit the release notes format.

To see a full list of commits added since last release, compare the revisions on GitHub.

Features:

  • 75dd48c38 Execution,Interface: Evict sandboxes based on their RSS (#2197)
  • 145c093b9 Execution,Interface: Charge idle canisters for full execution (#2183)
  • ec1cc659a Execution,Interface: Charge active canisters for full execution (#2180)
  • dbe869c88 Execution,Interface: Schedule and execute on low_wasm_memory_hook (#1877)
  • 944b8d0c3 Interface(ledger-core): Implement stable structures storable interface for Allowance (#2000)
  • f55a7562c Interface,Message Routing: Enable reject signals for requests (#1051)
  • c831d22ab Interface,Networking(https-outcalls): Enable H/2 support for outcalls (#2142)
  • b1ffe8292 Interface,Networking(call-v3): Enable the sync call handler on the NNS subnet (#2099)
  • 0fadddd59 Interface,Node(node-1474): Update config tool and create mac_address crate (#2123)
  • e31dc4c89 Owners(icrc-ledger-types): Implement stable structures storable interface for Account (#1998)
  • 00a3fb941 Owners(IDX): Use MODULE.bazel for http archive deps (#2128)

Bugfixes:

  • b7bf79f47 Execution,Interface: Do not update message metrics for messages that fail to prepay for execution (#2086)
  • 03dd6ee6d Interface(Ledger-Suite): renamed state machine tests (#2014)
  • c78e3549d Interface,Message Routing: Restore idle timeout functionality for XNetClientImpl (#2179)
  • 4b3c41b7c Interface,Networking: Remove unneed clones when serving requests from the grpc service (#2129)
  • 34b7182bf Interface,Networking(https-adapter): Bind the server socket before returning from start_server(). (#2150)
  • b545f6b10 Interface,Networking: remove async mutexes in the BTC adapter (#2116)
  • 6afff64b2 Owners(IDX): work around proc-macro2 non-determinism (#2176)
  • 1e22810de Node: add a temporary measure to remove existing API BN certs (#2146)
  • 1eae28c1e Node(recovery): Set up empty recovery directory in guestOS (#2131)
  • 2ce147d64 Node: allow read access to more hardware info for node_exporter (#2121)

Chores:

  • cce8a1797 Consensus,Interface(recovery): Improve error messages (#2122)
  • 0a5351777 Crypto,Interface: Upgrade core crates and use workspace version (#2111)
  • 5d1978e61 Execution,Interface,Message Routing: Drop CanisterQueue::QueueItem proto, part 2 (#1861)
  • 588ad7a46 Interface: upgrade rust version to 1.82 (#2137)
  • ee61d5074 Interface: Migrate all canisters to clap 4 (#2013)
  • 947217661 Interface: upgrade and use workspace version for quote and proc-macro2, remove ring where not used (#2095)
  • eadc89853 Interface,Networking: Upgrade opentelemetry crates (#2081)
  • d3def2417 Owners: Upgrade hyper, rustls and reqwest (#2162)
  • dff73c67c Node: Fixup services (#1837)
  • 5ca5d3c5b Node: Update Base Image Refs [2024-10-17-0809] (#2093)

Refactoring:

  • ecc5be798 Interface,Networking(http-handler): Remove async handler for the v3 call module (#2100)

Other changes:

  • edde85e8c Execution,Interface,Networking: “feat(): Adjust max number of cached sandboxes” (#2184)
  • 4e666d720 Interface,Networking: “feat(https-outcalls): Enable H/2 support for outcalls (#2142)” (#2159)

Full list of changes (including the ones that are not relevant to GuestOS) can be found on GitHub.

IC-OS Verification

To build and verify the IC-OS disk image, run:

# From https://github.com/dfinity/ic#verifying-releases
sudo apt-get install -y curl && curl --proto '=https' --tlsv1.2 -sSLO https://raw.githubusercontent.com/dfinity/ic/75dd48c38f296fc907c269263f96633fa8a29d0e/ci/tools/repro-check.sh && chmod +x repro-check.sh && ./repro-check.sh -c 75dd48c38f296fc907c269263f96633fa8a29d0e --guestos

The two SHA256 sums printed above from a) the downloaded CDN image and b) the locally built image, must be identical, and must match the SHA256 from the payload of the NNS proposal.

While not required for this NNS proposal, as we are only electing a new GuestOS version here, you have the option to verify the build reproducibility of the HostOS by passing --hostos to the script above instead of --guestos, or the SetupOS by passing --setupos.

2 Likes

Hello there!

We are happy to announce that voting is now open for a new IC release.
The NNS proposal is here: IC NNS Proposal 133799.

Here is a summary of the changes since the last release:

Release Notes for release-2024-10-23_03-07-hashes-in-blocks (8ac632c8e45181ac8e7ce04fbfe1e01e6cbd239b)

This release is based on changes since release-2024-10-23_03-07-base (75dd48c38f296fc907c269263f96633fa8a29d0e).

Please note that some commits may be excluded from this release if they’re not relevant, or not modifying the GuestOS image. Additionally, descriptions of some changes might have been slightly modified to fit the release notes format.

To see a full list of commits added since last release, compare the revisions on GitHub.

Features:

  • 8ac632c8e Interface,Networking, Consensus: Enable the hashes-in-blocks feature

IC-OS Verification

To build and verify the IC-OS disk image, run:

# From https://github.com/dfinity/ic#verifying-releases
sudo apt-get install -y curl && curl --proto '=https' --tlsv1.2 -sSLO https://raw.githubusercontent.com/dfinity/ic/8ac632c8e45181ac8e7ce04fbfe1e01e6cbd239b/ci/tools/repro-check.sh && chmod +x repro-check.sh && ./repro-check.sh -c 8ac632c8e45181ac8e7ce04fbfe1e01e6cbd239b --guestos

The two SHA256 sums printed above from a) the downloaded CDN image and b) the locally built image, must be identical, and must match the SHA256 from the payload of the NNS proposal.

While not required for this NNS proposal, as we are only electing a new GuestOS version here, you have the option to verify the build reproducibility of the HostOS by passing --hostos to the script above instead of --guestos, or the SetupOS by passing --setupos.

2 Likes

Hello there!

We are happy to announce that voting is now open for a new IC release.
The NNS proposal is here: IC NNS Proposal 133800.

Here is a summary of the changes since the last release:

Release Notes for release-2024-10-23_03-07-ubuntu20.04 (a6ef5933e981873fdbdbbed4cd1eae02e2917aa6)

This release is based on changes since release-2024-10-23_03-07-base (75dd48c38f296fc907c269263f96633fa8a29d0e).

Please note that some commits may be excluded from this release if they’re not relevant, or not modifying the GuestOS image. Additionally, descriptions of some changes might have been slightly modified to fit the release notes format.

To see a full list of commits added since last release, compare the revisions on GitHub.

Other changes:

  • a6ef5933e Node: Revert “chore: Fixup services (#1837)”
  • f471ef26e Node: Revert “fix(): allow read access to more hardware info for node_exporter (#2121)”
  • 6027218e9 Node: Revert “feat: Upgrade GuestOS to 24.04 (#938)”

IC-OS Verification

To build and verify the IC-OS disk image, run:

# From https://github.com/dfinity/ic#verifying-releases
sudo apt-get install -y curl && curl --proto '=https' --tlsv1.2 -sSLO https://raw.githubusercontent.com/dfinity/ic/a6ef5933e981873fdbdbbed4cd1eae02e2917aa6/ci/tools/repro-check.sh && chmod +x repro-check.sh && ./repro-check.sh -c a6ef5933e981873fdbdbbed4cd1eae02e2917aa6 --guestos

The two SHA256 sums printed above from a) the downloaded CDN image and b) the locally built image, must be identical, and must match the SHA256 from the payload of the NNS proposal.

While not required for this NNS proposal, as we are only electing a new GuestOS version here, you have the option to verify the build reproducibility of the HostOS by passing --hostos to the script above instead of --guestos, or the SetupOS by passing --setupos.

2 Likes

Thanks for this release @DRE-Team. I’ve skimmed through with the little time that I’m inclined to commit to this (since leaving CodeGov). One thing jumped out at me - there’s very little context provided for the following commit (at least that I could see - though I am currently sat in a park with sunshine, glare, and a mobile device :sun_with_face:) →

The commit message and associated pull request are focused on the what of the change and not the why.

The subnets that should not serve synchronous responses to v3 update calls… The list contains all system subnets

There was an explicit decision to exclude system subnets from serving synchronous responses in the past. Would it be possible to get some commentary on why that was and why it’s no longer considered appropriate (hence enabling it on the NNS subnet with this release)?

Maybe reviewers that are able to commit more time to a review will have an answer :slightly_smiling_face:

Proposal 133798: Adopt

I have reviewed all commits within proposal 133798 and in my opinion they all look fine.
I have also successfully run the build verification script, so I have voted to adopt the proposal.

Full review:

Features:

  • 75dd48c38 Execution,Interface: Evict sandboxes based on their RSS (#2197)
    Review: Looks fine + matches description
    Notes: Evicts sandbox processes based on their resident set size (the amount of memory allocated to their process) rather than evicting a fixed number each time. This allows for more sandbox processes while still keeping a limit on their total memory usage.

  • 145c093b9 Execution,Interface: Charge idle canisters for full execution (#2183)
    Review: Looks fine + matches description
    Notes: Modifies the scheduler so that at the end of each round it charges idle canisters which were scheduled ahead of the first non-idle canister the same as if they were fully executed. This reduces their priority in subsequent rounds allowing other canisters to appear earlier in the scheduler’s priority.

  • ec1cc659a Execution,Interface: Charge active canisters for full execution (#2180)
    Review: Looks fine + matches description
    Notes: Modifies the scheduler so that at the end of each round it charges canisters which were fully executed (they have no more messages or were scheduled first on a core).

  • dbe869c88 Execution,Interface: Schedule and execute on low_wasm_memory_hook (#1877)
    Review: Looks fine + matches description
    Notes: If on_low_wasm_memory_hook_status is Ready then when determining which task to run next the scheduler will pick the canister_on_low_wasm_memory hook and set the status to Executed.

  • 944b8d0c3 Interface(ledger-core): Implement stable structures storable interface for Allowance (#2000)
    Review: Looks fine + matches description
    Notes: Implements Storable for Allowance<Tokens> allowing it to be stored in stable memory within a StableBTreeMap.

  • f55a7562c Interface,Message Routing: Enable reject signals for requests (#1051)
    Review: Looks fine + matches description
    Notes: Starts pushing reject signals for messages which failed to be inducted iff the state certification version is V19, for lower version it continues to push reject responses along with accept signals.

  • c831d22ab Interface,Networking(https-outcalls): Enable H/2 support for outcalls (#2142)
    Review: Looks fine + matches description
    Notes: Updates the HttpsConnectorBuilder used by the HttpOutcalls adapter to use enable_all_versions rather than enable_http1, enabling it to support both HTTP1 and HTTP2.

  • b1ffe8292 Interface,Networking(call-v3): Enable the sync call handler on the NNS subnet (#2099)
    Review: Looks fine + matches description
    Notes: Updates enable_synchronous_call_handler_for_v3_endpoint to always return true, meaning that the synchronous V3 endpoint will now be enabled on all subnets.

  • 0fadddd59 Interface,Node(node-1474): Update config tool and create mac_address crate (#2123)
    Review: Looks fine + matches description
    Notes: Moves mgmt_mac out of ICOSDevSettings and into ICOSSettings and reads it from the deployment settings. Also moves the mac_address helper functions into their own package to avoid a circular dependency now that the types package references mac_address::FormattedMacAddress.

  • e31dc4c89 Owners(icrc-ledger-types): Implement stable structures storable interface for Account (#1998)
    Review: Looks fine + matches description
    Notes: Implements Storable for icrc_ledger_types::Account allowing it to be stored in stable memory within a StableBTreeMap.

  • 00a3fb941 Owners(IDX): Use MODULE.bazel for http archive deps (#2128)
    Review: Looks fine + matches description
    Notes: Moves config which specifies the download paths for external dependencies of the http archives from WORKSPACE.bazel to MODULE.bazel as part of the work to make the repo agnostic to Bazel download paths.

Bugfixes:

  • b7bf79f47 Execution,Interface: Do not update message metrics for messages that fail to prepay for execution (#2086)
    Review: Looks fine + matches description
    Notes: Adds the zero_instruction_messages metric which is incremented each time a message uses zero instructions due to being unable to pre-pay for execution.

  • 03dd6ee6d Interface(Ledger-Suite): renamed state machine tests (#2014)
    Review: Looks fine + matches description
    Notes: Simply renamed ic_icrc1_ledger_sm_tests to ic_ledger_suite_state_machine_tests.

  • c78e3549d Interface,Message Routing: Restore idle timeout functionality for XNetClientImpl (#2179)
    Review: Looks fine + matches description
    Notes: Adds a call to Client::builder::pool_timer(TokioTimer::new()) which is required by the newer version of Tokio to make the pool idle timeout functionality work.

  • 4b3c41b7c Interface,Networking: Remove unneed clones when serving requests from the grpc service (#2129)
    Review: Looks fine + matches description
    Notes: Removes some calls to clone in the Bitcoin adapter by working with references rather than owned values.

  • 34b7182bf Interface,Networking(https-adapter): Bind the server socket before returning from start_server(). (#2150)
    Review: Looks fine + matches description
    Notes: Within the HTTP outcalls adapter, ensures the server is ready on the main thread before calling rt_handle.spawn and returning from start_server.

  • b545f6b10 Interface,Networking: remove async mutexes in the BTC adapter (#2116)
    Review: Looks fine + matches description
    Notes: Within the Bitcoin adapter, replaces usages of the async tokio::sync::Mutex with the synchronous std::sync::Mutex.

  • 6afff64b2 Owners(IDX): work around proc-macro2 non-determinism (#2176)
    Review: Looks fine + matches description
    Notes: Applies a patch to rules_rust to work around the fact that proc_macro2 by default generates a dep-info file which references the build location, breaking determinism.

  • 1e22810de Node: add a temporary measure to remove existing API BN certs (#2146)
    Review: Looks fine + matches description
    Notes: Adds some temporary code to clear the existing certificates from the boundary nodes now that they have switched from HTTP- to the ALPN-ACME challenge.

  • 1eae28c1e Node(recovery): Set up empty recovery directory in guestOS (#2131)
    Review: Looks fine + matches description
    Notes: Sets up an empty recovery directory within GuestOS so that operators performing recovery don’t need to remember to first create one.

  • 2ce147d64 Node: allow read access to more hardware info for node_exporter (#2121
    Review: Looks fine + matches description
    Notes: Gives the prometheus node_exporter more read access to device info.

Chores:

  • cce8a1797 Consensus,Interface(recovery): Improve error messages (#2122)
    Review: Looks fine + matches description
    Notes: Improves the messages and formatting of some error messages to make them more readable.

  • 0a5351777 Crypto,Interface: Upgrade core crates and use workspace version (#2111)
    Review: Looks fine + matches description
    Notes: Bumps some dependencies to later versions and switches packages to use the workspace versions.

  • 5d1978e61 Execution,Interface,Message Routing: Drop CanisterQueue::QueueItem proto, part 2 (#1861)
    Review: Looks fine + matches description
    Notes: Drops the old CanisterQueue proto representation now that everything has migrated to the new version.

  • 588ad7a46 Interface: upgrade rust version to 1.82 (#2137)
    Review: Looks fine + matches description
    Notes: Bumps the Rust version from 1.81.0 to 1.82.0 and fixes some clippy issues raised by the newer version.

  • ee61d5074 Interface: Migrate all canisters to clap 4 (#2013)
    Review: Looks fine + matches description
    Notes: Updates all usages of clap to use the workspace version of 4.5.19.

  • 947217661 Interface: upgrade and use workspace version for quote and proc-macro2, remove ring where not used (#2095)
    Review: Looks fine + matches description
    Notes: Bumps some dependencies, removes some unused dependencies, reorders some dependencies to be ordered alphabetically, and updates some dependencies to use the workspace versions.

  • eadc89853 Interface,Networking: Upgrade opentelemetry crates (#2081)
    Review: Looks fine + matches description
    Notes: Bumps more dependencies.

  • d3def2417 Owners: Upgrade hyper, rustls and reqwest (#2162)
    Review: Looks fine + matches description
    Notes: Bumps even more dependencies.

  • dff73c67c Node: Fixup services (#1837)
    Review: Looks fine + matches description
    Notes: I’m not actually 100% sure what this change is doing but as far as I can tell it just cleans up a load of the service definition files for the services run by nodes.

  • 5ca5d3c5b Node: Update Base Image Refs [2024-10-17-0809] (#2093)
    Review: Looks fine + matches description
    Notes: Updates the IC-OS image references.

Refactoring:

  • ecc5be798 Interface,Networking(http-handler): Remove async handler for the v3 call module (#2100)
    Review: Looks fine + matches description
    Notes: Removes the async handler for the V3 endpoint now that all subnets have the synchronous version enabled.

Other changes:

  • edde85e8c Execution,Interface,Networking: “feat(): Adjust max number of cached sandboxes” (#2184)
    Review: Looks fine + matches description
    Notes: Bumps DEFAULT_MAX_SANDBOX_COUNT from 1000 to 2000.

  • 4e666d720 Interface,Networking: “feat(https-outcalls): Enable H/2 support for outcalls (#2142)” (#2159)
    Review: Looks fine + matches description
    Notes: Reverts c831d22ab because it was causing tests to fail.

Proposal 133799: Adopt

This proposal is the same as proposal 133788 except with the HASHES_IN_BLOCKS_FEATURE_ENABLED set to true.
I have run the build verification script successfully and so have voted to adopt this proposal.

Proposal 133800: Adopt

This proposal is the same as proposal 133788 except a few commits have been reverted in case they cause issues and this version is quickly needed.
I have run the build verification script successfully and so have voted to adopt this proposal.

4 Likes
Do not click on this or it will expand !
  • Copy pasta script to build and verify the IC-OS from proposal summary throws a tantrum.

Running it straight askes to be run inside a git repo.


Cloning the IC and running it again from within fixes this.

A quick peak at the script doesn’t seem to show any relevant changes.

Anyways while running the script multiple times will leave the following results here since it is not required going forward to check the matching hash except for GuestOS, I still did it for all in one run not with the use of separate flags.



As can bee seen for base all three match but SetupOS still fails for “hashes-in-blocks” and “ubuntu20.4” releases. At least it does on my Intel machine.

1 Like

Proposal #133798

Vote: Adopted
Reason: Builds fine and the hash matches.

Reviews

Refactoring:
ecc5be798 Interface,Networking(http-handler): Remove async handler for the v3 call module (#2100)
Done by removing new_asynchronous_call_service_router and async_v3_handler since the synchronous call handler is enabled now on all subnets for V3 calls.

Other changes:
edde85e8c Execution,Interface,Networking: “feat(): Adjust max number of cached sandboxes” (#2184)
Doubles the DEFAULT_MAX_SANDBOX_COUNT .
4e666d720 Interface,Networking: “feat(https-outcalls): Enable H/2 support for outcalls (#2142)” (#2159)
Reverting the previous commit c831d22 because of failing tests.

1 Like

Proposal #133799

Vote: Adopted
Reason: Builds fine and the hash matches.

Reviews

The only one additional change is
Features:
8ac632c8e Interface,Networking, Consensus: Enable the hashes-in-blocks feature.
This is done by setting HASHES_IN_BLOCKS_FEATURE_ENABLED feature falg to true in order to “strip all ingress messages from blocks, before sending them to peers”. Matches description.

2 Likes

Proposal #133800

Vote: Adopted
Reason: Builds fine and the hash matches.

Reviews

Other changes:
a6ef5933e Node: Revert “chore: Fixup services (#1837)”
Done by reverting commit dff73c6 from last week.
f471ef26e Node: Revert “fix(): allow read access to more hardware info for node_exporter (#2121)”
Done by reverting commit 2ce147d that added more read access to device info for node_exporter (allowing reading udev state data from /run/udev/data, reading /proc/pressure and reading under mount points with mnt_t) so they are now removed.
6027218e9 Node: Revert “feat: Upgrade GuestOS to 24.04 (#938)”
Done by reverting commit 87ed927.
All three match their description.

2 Likes

proposal - 133798

Vote: ADOPT

Reason:

The build was successful, and all the commit descriptions matched the changes in the code. I only have one piece of feedback: there’s a typo in a function name, but that’s about it. I have no concerns after reviewing each commit.

Hash Match: MATCH

2 Urls: MATCH

Feedback:

Proposer Check: MATCH

Overall Summary:

The updates introduce better memory management by adding an rss field to track memory usage of sandbox processes, increasing the DEFAULT_MAX_SANDBOX_COUNT to 5000 for more efficient evictions. Scheduler improvements ensure that canisters are correctly tracked upon completing their execution, optimizing resource allocation. Stable storage integration for the Allowance struct enhances serialization, ensuring data consistency during migrations.

Commits Summary

Commits:

Features:

75dd48c38
Adds the num-traits dependency to cargo. “Sandbox RSS" refers to the total resident set size (RSS) of memory used by sandbox processes, with a set limit (in this code, 50 GiB) after which sandbox processes may be evicted to manage memory efficiently. Sandbox process is terminated or removed from memory when it exceeds DEFAULT_MAX_SANDBOXES_RSS. DEFAULT_MAX_SANDBOX_COUNT went from 500 to 5k.

The updated code introduces an rss field in EvictionCandidate to track memory usage, allowing the evict function to consider both idle time and memory limits (max_sandboxes_rss) when selecting candidates for eviction.

145c093b9
charge_idle_canisters iterates through canisters in a specific order and completes the execution of any canisters in an idle state, marking them as fully executed, while skipping those in the process of code installation or already actively executing tasks.

ec1cc659a
Now inner_round of scheduler returns round_fully_executed_canister_ids.

finish_canister_execution marks a canister as fully executed if it has completed all pending executions or is the first canister in the iteration, recording the round and adding it to a list of fully executed canister IDs.

finish_round charges canisters that completed full executions by updating their priority_credit and distributes any remaining compute capacity across all canisters, adjusting their accumulated_priority based on their compute allocation and available free capacity.

The values for charging seem reasonable and scalable.

dbe869c88
The checks for on_low_wasm_memory_hook_status in the front and pop_front methods ensure that memory management tasks are prioritized in the canister’s execution.

The OnLowWasmMemory hook allows a canister to respond when its memory usage reaches a critical threshold, enabling the developer to perform cleanup or memory optimization tasks.

944b8d0c3
Integrates stable storage by implementing the Storable trait for the Allowance struct using candid::Nat and ic_stable_structures, enabling serialization compatible with stable memory. Additionally, it modifies the serialization logic to exclude the arrived_at field, ensuring the Allowance data is correctly managed during the migration to stable structures.

It matches the description.

f55a7562c
The recent changes in the StreamHandler involve updating how message induction errors are handled. The induct_message_impl method now returns Option<(RejectReason, RequestOrResponse)> instead of Result, refining how different outcomes like drops, rejections, and reroutes are managed. Additionally, the handling of legacy certification versions has been adjusted, including updates to tests for certification version behaviors and deprecation of older logic tied to versions below V19. There are some test specific to canister migration to different subnets. Test verifies that after a canister is migrated between subnets, all pending calls conclude properly without leaving any hanging calls, ensuring that the message routing and migration process works as expected.

c831d22ab
HTTP/2 Support enabled.

Test verifies that the outcalls adapter correctly supports HTTP/2 and HTTP/1.1 protocols by negotiating the appropriate ALPN (Application-Layer Protocol Negotiation) protocol during a TLS handshake. It sets up a test server with specified ALPN protocols, sends a request, and checks if the response has a status of OK only when the negotiated protocol matches the expected HTTP version.

b1ffe8292
Now we always allows synchronous responses to v3 update calls regardless of the subnet. This matches the commit description.

0fadddd59
New mac_address module, adding functionality for handling MAC addresses, including parsing, formatting, and deriving values dynamically when not provided. This refactoring separates MAC address handling from other network components. Additionally, ICOSSettings now requires mgmt_mac and deployment_environment as fields.

e31dc4c89
Storable trait implementation for the Account struct, enabling efficient serialization and deserialization with a defined MAX_SERIALIZATION_LEN for bounded storage size. It also adds dependencies on ic_stable_structures and includes functionality for converting Account instances into byte representations and reading them back.

00a3fb941
Moving various http_archive, http_file, and other dependencies from WORKSPACE.bazel into MODULE.bazel. This restructuring helps organize external dependencies and configurations more cleanly, centralizing them in MODULE.bazel while reducing clutter in the WORKSPACE.bazel file.

Bugfix:

b7bf79f47
New metric scheduler_zero_instruction_messages.
Check for non-zero message counts before recording execution duration, and if no messages are processed, it stops the timer without recording and increments a zero_instruction_messages metric which matches commit description.

03dd6ee6d
Change ic-icrc1-ledger-sm-tests to ic-ledger-suite-state-machine-tests, updating references across various Bazel and Cargo files to reflect this change. Dependencies and test paths were adjusted accordingly to support the renamed modules.

c78e3549d
Addition of TokioExecutor and TokioTimer imports from hyper_util::rt. Fix in http_client add pool_timer

4b3c41b7c
Refactors how VecDeque queues are managed in the get_successor_blocks and get_next_headers functions, changing from a vector of BlockHash copies to using references (&BlockHash). This change optimizes memory usage by reducing the need to clone block hashes during the breadth-first search and header retrieval processes.

34b7182bf
Refactors the handling of the incoming connection source by creating a boxed Stream for IncomingSource::Path and IncomingSource::Systemd. So incoming is setup sync before rt_handle.spawn.

b545f6b10
Replace asynchronous locks with synchronous locks using Mutex instead of tokio::sync::Mutex in the Bitcoin adapter. This aligns with the commit asynchronous mutex is more expensive than an ordinary mutex.

6afff64b2
Workaround for build determinism issues with proc-macro2 by checking if the proc_macro2.d file exists in the output directory and removing it if it does. Matches commit description.

1e22810de
Remove certs.

1eae28c1e
Add /var/lib/ic/data/recovery into script creation to make it easier for administrator.

2ce147d64
Allow more read access to node_exporter_t. Expanding the health metrics of the system.

Chores:

cce8a1797
GracefulExpect trait, providing a custom expect_graceful method that prints user-friendly error messages before terminating, replacing the default expect calls throughout the recovery code. This change is applied across various modules like app_subnet_recovery, nns_recovery_failover_nodes, and nns_recovery_same_nodes. Matches description.

0a5351777
Several dependency updates, such as upgrading “paste” from version 1.0.14 to 1.0.15, “subtle” from 2.5.0 to 2.6.1, and “rsa” from 0.9.2 to 0.9.6. Many packages now use a workspace configuration for shared dependencies like “serde” and "subtle,”. Matches description.

5d1978e61
Removing the deprecated deprecated_queue field from the CanisterQueue message structure in protobuf definitions and corresponding Rust code. This matches the commit description.

588ad7a46
Updating the base image for the build and CI processes to a new version across multiple configuration files. Update Rust toolchain from version 1.81.0 to 1.82.0.

ee61d5074
Upgrade the clap dependency from version 3.2.25 to 4.5.19 and integrating it into the workspace. This matches the commit description.

947217661
Update of the proc-macro2 crate from version 1.0.85 to 1.0.88 and the quote crate from 1.0.35 to 1.0.37. Removal of ring dep which matches the commit description.

eadc89853
Upgrade async-trait from version 0.1.81 to 0.1.83, and async-stream from 0.3.5 to 0.3.6. Several updates are also made to OpenTelemetry dependencies. Removal of older versions of hyper-timeout and opentelemetry-prometheus.

d3def2417
Matches description of upgrade hyper, rustls and reqwest.

dff73c67c
Adjusting the configuration and setup of systemd services within the boundary and guest operating systems. Adjusting the order of After and Before targets for services like node_exporter, setup-hostname, and vsock-agent, as well as renaming and restructuring systemd service files. Adjustments to file paths and labels for various setup scripts.

5ca5d3c5b
Matches description of updating base container image references.

Refactoring:

ecc5be798
Removal of async_v3_handler, new_asynchronous_call_service_router. Now we just have sync call_v3_router.

Other:

edde85e8c
Increase DEFAULT_MAX_SANDBOX_COUNT to 2_000

4e666d720
Similar to reviewed c831d22ab

proposal - 133799

Vote: ADOPT

Reason:

Build is successful and code change matches description.

Hash Match: MATCH

2 Urls: MATCH

Feedback: NONE

Proposer Check: MATCH

Commits:

8ac632c8e
Enables feature flag HASHES_IN_BLOCKS_FEATURE_ENABLED

proposal - 133800

Vote: ADOPT

Reason:

Build is successful and code change matches description. However, it is a bit concerning that we are reverting commits that we just added in proposal - 133798.

Hash Match: MATCH

2 Urls: MATCH

Feedback: NONE

Proposer Check: MATCH

Commits:

[a6ef5933e]
Reverting commit dff73c67c

[f471ef26e]
Reverting commit 2ce147d64

[6027218e9]
Reverting commit 87ed927

3 Likes

Proposal 133798

Summary

  1. Vote: Adopt
  2. Hash: All the hashes match
  3. Reasons to adopt: The release notes match the commits and the code changes. Builds successfully and all the hashes match

Commits Review

Features

75dd48c38
Summary: Evict sandboxes based on RSS
Notes
Allows increasing the sandbox process without risking OOM since the total RSS size of the sandbox is limited.
This PR also increases the number of sandbox processes to 5K from 2K
DEFAULT_MAX_SANDBOX_COUNT to set to 5K
Review: Code changes matches the commit message.

145c093b9
Summary: Charge idle canister for full execution
Notes:
Idle canister in front of the round schedule should be marked as fully executed which helps in rotating the round faster.
This is cherry pick of previous reverted commit #1806
Review:
Code changes match the commit message

ec1cc659a
Summary: Charge active canister for full execution
Notes:
Revert of #1782
Charges canister for full execution, total points charged is evenly distributed among all the canisters.
Review:
Code changes match the commit message

dbe869c88
Summary: Enables OnLowWasmMemory hook
Notes:
This PR enables OnLowWasmMemory feature and adds test cases for it
Review:
Code changes match the commit message

944b8d0c3
Summary: Implements stable structures storable interface for Allowance
Notes:
Adds functionality for ic-stable-structures storable interface and adds test cases for the same.
Review:
Code changes match the commit message

f55a7562c
Summary: Enables reject message for requests
Notes:
This change keeps generating reject response for all the request below CertificationVersion::V19. Old test cases are modified to create new test cases and old test cases use naming convention legacy_ prefix. Also adds one new state machine test for a canister migration to check all
calls are concluded for this scenario.
Review:
Code changes match the commit message

c831d22ab
Summary: Enables H/2 support for outcall
Notes:
This PR enables H/2 support for outcalls updates dev dependencies and adds test cases.
Review:
Code changes match the commit message

b1ffe8292
Summary: Enables sync call handler on NNS subnets
Notes:
This PR updates enable_synchronous_call_handler_for_v3_endpoint method to always return true so the synchronous v3 will be enabled for all the subnets.
Review:
Code changes match the commit message

0fadddd59
Summary: Updates config tool and create mac_address
Notes:
This PR adds mgmt_mac to ICOSSettings
renames deployment_environment
To fix cycle dependency between network and config it creates mac_address crate
Review:
Code changes match the commit message

e31dc4c89
Summary: Implements stable structure storable interface for Accounts
Notes:
Adds functionality for ic-stable-structures storable interface for Accounts and adds test cases for the same.
Review:
Code changes match the commit message

00a3fb941
Summary Use MODULE.bazel for http archive deps
Notes:
Moves http archives configs from WORKSPACE.bazel to MODULE.bazel
Review:
Code changes match the commit message

Bugfixes

b7bf79f47
Summary Fails prepay metric
Notes:
Updates per-message metrics to not count message which fail to pre-pay for execution.
Adds a new counter zero_instruction_messages that increments each time a scheduled message fails to actually execute.
Review:
Code changes match the commit message

03dd6ee6d
Summary Renamed state machine tests
Notes:
renames ic-icrc1-ledger-sm-tests to ic-ledger-suite-state-machine-tests
Review:
Code changes match the commit message

c78e3549d
Summary Restore idle timeout functionality for XNetClientImpl
Notes:
update of hyper-util from 0.14.18 to 0.1.8 requires an additional call pool_timer() to set idle timeout timer to drop idle connections from the pool.
Review:
Code changes match the commit message

4b3c41b7c
Summary Remove unneed clones when serving requests from the grpc service
Notes:
removes/updates from service request calls from get_successors_handler.rs
Review:
Code changes match the commit message

34b7182bf
Summary Bind the server socket before returning from start_server()
Notes:
This PR adds code to bind the server socket before returning from start_server() method.
Review:
Code changes match the description

b545f6b10
Summary remove async mutexes in the BTC adapter
Notes:
In Bitcoin adapter updates the async mutex call to synchronous mutex call
Review:
Code changes match the description

6afff64b2
Summary work around for proc-macro2
Notes:
There were some build issues in proc-macro2 Bazel build. This PR provides a work around for rules_rust to address those build issues.
Review:
Code changes match the description

1e22810de
Summary temporary fix to remove existing certificates of the API BN
Notes:
Provides temporary fix to clean-up pre-existing certificates to address the switch from the HTTP- to the ALPN-ACME.
Review:
Code changes match the description

1eae28c1e
Summary Set up empty recovery directory in guestOS
Notes:
This PR adds code to create /var/lib/ic/data/recovery empty directory with admin permission for recovery. With this change the operator does not need to create an empty directory and update the permissions manually.
Review:
Code changes match the description

2ce147d64
Summary allow read access to more hardware info for node_exporter
Notes:
This PR adds code to allow read access for the following

  • /run/udev/data
  • /proc/pressure
  • mount points with mnt_t

Review:
Code changes match the description

Chores

cce8a1797
Summary Improves error messages
Notes:
Updates various error messages.
Review:
Code changes match the description

0a5351777
Summary upgrade core crates and use workspace version
Notes:
Updates versions and checksum for various dependencies and updates packages to use workspace version
Review:
Code changes match the description

5d1978e61
Summary Drop CanisterQueue::QueueItem proto
Notes:
removes the deprecated canister queue references and keep newer u64 representation.
Review:
Code changes match the description

588ad7a46
Summary upgrade rust version to 1.82
Notes:
Upgrades rust version from 1.81.0 to 1.82.0
disables the TLA tool per instruction since it was running out of memory
changes as per clipyy lints
Review:
Code changes match the description

ee61d5074
Summary Migrates canister to clap4
Notes:
Updates all usage of clap from version: 3.2.25 to workspace
Review:
Code changes match the description

947217661
Summary Upgrade and use workspace version for quote and proc-macro2
Notes:
upgrades version for some dependencies and updates there checksum.
removes unused dependencies.
updates dependencies to use workspace versions
Review:
Code changes match the description

eadc89853
Summary upgrade opentelemetry crates
Notes:
upgrades some package versions, removes unused packages and upgrades opentelemetry version from 0.23.0 to 0.26.0
Review:
Code changes match the description

d3def2417
Summary upgrade hyper, rustls and reqwest
Notes:
upgrades hyper from version 1.4.1 to 1.5.0 and rustls from version 0.23.14 to 0.23.15 and reqwest from version 0.12.7 to 0.12.8
Review:
Code changes match the description

dff73c67c
Summary Fixup services
Notes:
some service level changes and some changes to config
Review:
Code changes match the description

5ca5d3c5b
Summary Updates Base Image Refs
Notes:
This PR updates several base container images
Review:
Code changes match the description

Refactoring

ecc5be798
Summary Remove async handler for v3 call
Notes:
Removes asynchronous v3 calls and enables synchronous v3 calls
Review:
Code changes match the description

Other changes

edde85e8c
Summary reverts commit 75c57bc
Notes:
Updates DEFAULT_MAX_SANDBOX_COUNT from 1_000 to 2_000
Review:
Code changes match the description

4e666d720
Summary
reverts commit c831d22
Notes:
This PR reverts c831d22 temporarily since manually triggered tests are failing.
Review:
Code changes match the description

Proposal 133799

Summary

  1. Vote: Adopt
  2. Hash: All the hashes match
  3. Reasons to adopt: The release notes match the commits and the code changes. Builds successfully and all the hashes match

Commits Review

Features

8ac632c8e
Summary
Enables hashes-in-blocks feature flag
Notes:
updates HASHES_IN_BLOCKS_FEATURE_ENABLED feature flag to true. When this feature flag is enabled we strip all ingress messages from blocks before sending them to peer. They are reconstructed on the receiver side by looking at the ingress messages in the pool and if the message is missing from the ingress pool it is fetched from the peers.
Review:
Code changes match the description

Proposal 133800

Summary

  1. Vote: Adopt
  2. Hash: All the hashes match
  3. Reasons to adopt: The release notes match the commits and the code changes. Builds successfully and all the hashes match

Commits Review

Other changes

a6ef5933e
Summary
reverts commit dff73c6
Notes:
reverts commit dff73c6: Fixup services
Review:
Code changes match the description

f471ef26e
Summary
reverts commit 2ce147d
Notes:
reverts commit 2ce147d: allow read access to more hardware info for node_exporter
Review:
Code changes match the description

6027218e9
Summary
reverts commit 87ed927
Notes:
reverts commit 87ed927: Upgrade GuestOS to 24.04
Review:
Code changes match the description

1 Like

Proposal 133798

Vote: ADOPTED.
Reason: All reviewed commits match their description and hashes match.

Review:
For the Execution and Runtime layers, the main changes introduced in this proposal are:

  • Improving the sandbox processes eviction logic to increase the number of the sandbox processes
  • Adding the improvements to the canisters scheduling to the main release for all the subnets
  • Concluding the OnLowWasmMemory hook feature

I’ve reviewed all the commits listed in the proposal, for the Execution and Runtime layers:
75dd48c38:
Adds the max_sandboxes_rss (default value: 50 GiB) and removes the min_sandbox_count fields from the Config struct in the embedders configuration.
Changes the logic of the evict method inside the sandbox_process_eviction to add a canister to the returned list of evicted canisters. If we find a canister that has not been used for a long time, the evicted canisters are below the max count and the total RSS is below the max sandboxes RSS + the total RSS sum of the evicted canisters until now, we stop evicting the canisters. For each canister, the RSS is taken from the newly introduced rss field inside the SanboxProcessStats. This field is updated in the newly introduced update_sandbox_processes_rss function, that is invoked in the monitor_and_evict_sandbox_processes method of the SandboxedExecutionController struct. The total RSS passed to the evict function is calculated as the sum of the RSS of each active sandbox in the newly introduced function total_sandboxes_rss.
Added a comment on GitHub.

145c093b9:
Introduces the same changes introduced in the proposal 133450.

ec1cc659a:
Introduces the same changes introduced in the proposal 133450.

dbe869c88:
Changes the is_next_method_chosen function in the scheduler to return true if the task in front of the queue is a hook. In order to determine if the task is a hook, the is_hook method is introduced in the ExecutionTask enum. The is_hook method returns true only if the enum variant is OnLowWasmMemory.
Changes the front and pop_front methods of the TaskQueue struct to return the OnLowWasmMemory variant if the hook status is ready. It also changes the is_empty and len methods to account when the hook is ready.
Adds the canister_on_low_wasm_memory to the valid functions exported by the canister’s Wasm.
This commits concludes the OnLowWasmMemory hook feature.

b7bf79f47:
Adds the zero_instruction_messages field (counter) to the SchedulerMetrics, which is incremented when the instructions_used is 0 in the execute_canisters_on_thread function.

5d1978e61:
Removes the deprecated_queue field from the CanisterQueue in the protobuf definition.

edde85e8c:
Increases the DEFAULT_MAX_SANDBOX_COUNT to 2000. This change is overwritten by the commit 75dd48c38, which sets it to 5000.


Proposal 133799

Vote: ADOPTED.
Reason: All reviewed commits match their description and hashes match.

Review:
This release is the same as the one in 133798 with the addition of the commit 8ac632c8e, which matches its description.


Proposal 133800

Vote: ADOPTED.
Reason: All reviewed commits match their description and hashes match.

Review:
This release is the same as the one in 133798 with the addition of the following commits:
a6ef5933e:
Matches description.

f471ef26e:
Matches description.

6027218e9:
Matches description.

2 Likes

Proposal 133798

Summary

  1. Vote: Adopt
  2. Hash: Hashes match
  3. Reasons to adopt: Builds fine + hashes match + release notes match the commits
Commits Review

Features:

  1. [75dd48c38]
    Summary:
    Evict sandboxes based on their RSS.
    Notes:
    Increase sandbox process to 5k, OOM will be prevented by limiting the total RSS size of sandbox processes.
    Review:
    Description matches.

  2. [145c093b9]
    Summary:
    Charge idle canisters for full execution.
    Notes:
    Verified that the idle canisters in the front of the queue must be marked as fully executed for better performance.
    Review:
    Description matches.

  3. [ec1cc659a]
    Summary:
    Charge active canisters for full execution.
    Notes:
    Calculate new execution cores for each fully executed canister.
    Review:
    Description matches.

  4. [dbe869c88]
    Summary:
    Schedule and execute on low_wasm_memory_hook.
    Notes:
    OnLowWasmMemory feature is enabled by implementing the hook.
    Review:
    Description matches.

  5. [944b8d0c3]
    Summary:
    Implement stable structures storable interface for Allowance.
    Notes:
    Implements Storable for Allowance using ic_stable_structures`. Tests have also been added.
    Review:
    Description matches.

  6. [f55a7562c]
    Summary:
    Enable reject signals for requests.
    Notes:
    Generates reject responses for versions below CertificationVersion::V19 and enables reject signals for requests for CertificationVersion::V19 and above.
    Review:
    Description matches.

  7. [c831d22ab]
    Summary:
    Enable H/2 support for outcalls.
    Notes:
    H/2 support for outcalls enabled and dependencies updated.
    Review:
    Description matches.

  8. [b1ffe8292]
    Summary:
    Enable the sync call handler on the NNS subnet.
    Notes:
    Added sync call handler on the Network Nervous System (NNS) subnet. Synchronous v3 will be enabled as enable_synchronous_call_handler_for_v3_endpoint will always return true.
    Review:
    Description matches.

  9. [0fadddd59]
    Summary:
    Update config tool and create mac_address crate.
    Notes:
    Some config revamp: Added mgmt_mac to ICOSSettings and renamed deployment_environment. Fix cyclic dependency between network and config by creating mac_address crate.
    Review:
    Description matches.

  10. [e31dc4c89]
    Summary:
    Implement stable structures storable interface for Account.
    Notes:
    Implements Storable for Account using ic_stable_structures`.
    Review:
    Description matches.

  11. [00a3fb941]
    Summary:
    Use MODULE.bazel for http archive deps
    Notes:
    Port http archives configs from WORKSPACE.bazel to MODULE.bazel
    Review:
    Description matches.

Bugfixes:

  1. [b7bf79f47]
    Summary:
    Do not update message metrics for messages that fail to prepay for execution.
    Notes:
    Add a new counter zero_instruction_messages that increments each time a scheduled message fails to execute instead of updating the per-message metrics to count messages that failed execution.
    Review:
    Description matches.

  2. [03dd6ee6d]
    Summary:
    Renamed state machine test.
    Notes:
    Rename icrc1 state machine tests from ic-icrc1-ledger-sm-tests to ic-ledger-suite-state-machine-tests.
    Review:
    Description matches.

  3. [c78e3549d]
    Summary:
    Restore idle timeout functionality for XNetClientImpl.
    Notes:
    Add pool_timer() call to fix idle timeout functionality as hyper-util was upgraded from 0.14.18 to 0.1.8.
    Review:
    Description matches.

  4. [4b3c41b7c]
    Summary:
    Remove unneeded clones when serving requests from the grpc service.
    Notes:
    Remove unused clones when processing grpc requests from from get_successors_handler.rs .
    Review:
    Description matches.

  5. [34b7182bf]
    Summary:
    Bind the server socket before returning from start_server().
    Notes:
    Implements code to bind the server socket before returning from start_server().
    Review:
    Description matches.

  6. [b545f6b10]
    Summary:
    Remove async mutexes in the BTC adapter.
    Notes:
    Use an ordinary mutex instead of an async mutex in BTC adapter as it is more expensive.
    Review:
    Description matches.

  7. [6afff64b2]
    Summary:
    Work around proc-macro2 non-determinism.
    Notes:
    Patch to fix determinism issues for theproc_macro2 Bazel build. rules_rust patch was added.
    Review:
    Description matches.

  8. [1e22810de]
    Summary:
    Add a temporary measure to remove existing API BN certs
    Notes:
    Remove existing certificates of the API BNs to address the switch from the HTTP- to the ALPN-ACME.
    Review:
    Description matches.

  9. [1eae28c1e]
    Summary:
    Set up empty recovery directory in guestOS.
    Notes:
    Set up an empty recovery directory at /var/lib/ic/data/recovery with admin permissions to simplify recovery in guestOS.
    Review:
    Description matches.

  10. [2ce147d64]
    Summary:
    Allow read access to more hardware info for node_exporter
    Notes:
    Provide more HW info to prometheus node_exporter, the following information will be exposed -

  • /run/udev/data
  • /proc/pressure
  • mount points with mnt_t
    Review:
    Description matches.

Chores:

  1. [cce8a1797]
    Summary:
    Improve error messages.
    Notes:
    Added more descriptive error messages.
    Review:
    Description matches.

  2. [0a5351777]
    Summary:
    Upgrade core crates and use workspace version.
    Notes:
    Update various crate/library versions.
    Review:
    Description matches.

  3. [5d1978e61]
    Summary:
    Drop CanisterQueue::QueueItem proto, part 2.
    Notes:
    Drop the deprecated CanisterQueue::QueueItem protobuf and instead use the u64 representation.
    Review:
    Description matches.

  4. [588ad7a46]
    Summary:
    Upgrade rust version to 1.82.
    Notes:
    Upgrade the rust version of the repo 1.81.0 to 1.82.0 and disable the TLA tool per instruction to prevent running out of memory.
    Review:
    Description matches.

  5. [ee61d5074]
    Summary:
    Migrate all canisters to clap 4.
    Notes:
    Upgrade all canisters to use the clap4 command-line argument parser for Rust.
    Review:
    Description matches.

  6. [947217661]
    Summary:
    Upgrade and use workspace version for quote and proc-macro2, remove ring where not used.
    Notes:
    Upgrade metaprogramming crate versions for quote and proc-macro2 and remove the cryptography library ring where it is unused.
    Review:
    Description matches.

  7. [eadc89853]
    Summary:
    Upgrade opentelemetry crates.
    Notes:
    Upgrade the collection of Rust libraries that implement the OpenTelemetry specification from 0.23.0 to 0.26.0.
    Review:
    Description matches.

  8. [d3def2417]
    Summary:
    Upgrade hyper, rustls and reqwest.
    Notes:
    Upgrade Network Operations crates: hyper (from version 1.4.1 to 1.5.0), rustls (from version 0.23.14 to 0.23.1) and request (from version 0.12.7 to 0.12.8).
    Review:
    Description matches.

  9. [dff73c67c]
    Summary:
    Fixup services.
    Notes:
    Fix services (config and service level)
    Review:
    Description matches.

  10. [5ca5d3c5b]
    Summary:
    Update Base Image Refs [2024-10-17-0809].
    Notes:
    Update base docker container image references.
    Review:
    Description matches.

Refactoring:

  1. [ecc5be798]
    Summary:
    Remove async handler for the v3 call module
    Notes:
    Remove the async handler for v3 as the synchronous call handler is enabled on all v3 subnets.
    Review:
    Description matches.

Other changes:

  1. [edde85e8c]
    Summary:
    “feat(): Adjust max number of cached sandboxes”
    Notes:
    Revert commit 75c57bc. The DEFAULT_MAX_SANDBOX_COUNT variable value is updated from 1_000 to 2_000
    Review:
    Description matches.

  2. [4e666d720]
    Summary:
    "feat(https-outcalls): Enable H/2 support for outcalls
    Notes:
    Revert commit c831d22 as manually triggered tests are failing after merging this feature.
    Review:
    Description matches.

Proposal 133799

Summary

  1. Vote: Adopt
  2. Hash: Hashes match
  3. Reasons to adopt: Builds fine + hashes match + release notes match the commits
Commits Review

Features:

  1. [8ac632c8e]
    Summary:
    Enable the hashes-in-blocks feature.
    Notes:
    Update the boolean HASHES_IN_BLOCKS_FEATURE_ENABLED to true, enabling the hashes-in-blocks feature. It will get all ingress messages from blocks. On the receiver side, it will reconstruct the blocks by referencing the ingress messages or fetching missing ingress messages.
    Review:
    Description matches.

Proposal 133800

Summary

  1. Vote: Adopt
  2. Hash: Hashes match
  3. Reasons to adopt: Builds fine + hashes match + release notes match the commits
Commits Review

Other changes:

  1. [a6ef5933e]
    Summary:
    Revert “chore: Fixup services”.
    Notes:
    Revert commit dff73c6: fixup services.
    Review:
    Description matches.

  2. [471ef26e]
    Summary:
    Revert “fix(): allow read access to more hardware info for node_exporter”.
    Notes:
    Revert commit reverts commit 2ce147d: allow read access to get more HW info for node exporter.
    Review:
    Description matches.

  3. [6027218e9]
    Summary:
    Revert "feat: Upgrade GuestOS to 24.04.
    Notes:
    Revert commit 87ed927: Update GuestOS version to 24.04.
    Review:
    Description matches.

1 Like

Proposal: 133798

Summary:

  1. Build Hash: Build has from the proposal, local build and CDN matches and is “d22128703032465790b971ff4487afe228f49084051994241939707fb602e612”.
  2. Summary: The release notes matches the code changes
  3. Vote: I vote to adopt the proposals

Detailed Review:

Features:

  • 75dd48c38 Execution,Interface: Evict sandboxes based on their RSS (#2197)
    Notes: This update introduces a new sandbox eviction process based on Resident Set Size (RSS) to manage memory more effectively. This approach ensures memory constraints are respected without impacting performance. The modified function now specifies a maximum RSS threshold, above which the least recently used sandboxes are removed first.
    Review: Code changes look good and match release notes.
  • 145c093b9 Execution,Interface: Charge idle canisters for full execution (#2183)
    Notes: These code changes enforces full execution for idle canisters that remain at the start of the scheduling round. This PR is a modified reintroduction of previously reverted changes.
    Review: Code changes look good and match release notes.
  • ec1cc659a Execution,Interface: Charge active canisters for full execution (#2180)
    Notes: These code changes are in the scheduler to charge the canisters which were fully executed and no messages to execute or were scheduled first on the core.
    Review: Matches release notes and code changes look good.
  • dbe869c88 Execution,Interface: Schedule and execute on low_wasm_memory_hook (#1877)
    Notes: canister_on_low_wasm_memory hook is introduced. Changes are made to the scheduler to execute the task in the task queue if it is a hook.
    Review: Code changes matches release notes and look good
  • 944b8d0c3 Interface(ledger-core): Implement stable structures storable interface for Allowance (#2000)
    Notes: This implements a stable structure Storable for Allowance in the ledger-core for approval feature.
    Review: Code changes looks good and matches release notes
  • f55a7562c Interface,Message Routing: Enable reject signals for requests (#1051)
    Notes: Code changes are made to send a reject signal for messages failed to induct msg (error or canister migrating) and state.metadata.certificateVersion is > 19. For lower version it will keep sending accept signal while sending reject response.
  • c831d22ab Interface,Networking(https-outcalls): Enable H/2 support for outcalls (#2142)
    Notes: Code changes are made to support all version of http outcalls which were previously only limited to Http1. Changes were made to adapter to enable all versions.
    Review: Code changes match release notes and look good.
  • b1ffe8292 Interface,Networking(call-v3): Enable the sync call handler on the NNS subnet (#2099)
    Notes: Function enable_synchronous_call_handler_for_v3_endpoint will now always return true, and hence sync call is enabled in all the case. The title of the PR could have been more informative here that it is now enabled for all subnets. Also the inline comments should be updated to reflect that.
    Review: Title of the PR could be more informative and accurate towards code changes. But changes looks good here.
  • 0fadddd59 Interface,Node(node-1474): Update config tool and create mac_address crate (#2123)
    Notes: The config tool is now updated to calculate mgmt_mac based on if the config_mac provided otherwise take the IPMI mac address. This mgmt_mac is now also moved under icos_settings and also typed to mac_address::mac_address::FormattedMacAddress;
    Review: Code changes look good and match release notes.
  • e31dc4c89 Owners(icrc-ledger-types): Implement stable structures storable interface for Account (#1998)
    Notes: The code changes implement stable structure Storable for Account and changes are made to icrc_ledger_types
    Review: Code changes look good and match release notes.
  • 00a3fb941 Owners(IDX): Use MODULE.bazel for http archive deps (#2128)
    Moves the config to pull dependencies from WOKSPACE.bazel to MODULE.bazel to make it more agnostic.

Bugfixes:

  • b7bf79f47 Execution,Interface: Do not update message metrics for messages that fail to prepay for execution (#2086)
    Notes: zero_instruction_messages metric is added to count message which could not be executed probably because the canisters were unable to prepay.
    Review: Code changes matches release notes
  • 03dd6ee6d Interface(Ledger-Suite): renamed state machine tests (#2014)
    Notes: ic_icrc1_ledger_sm_tests tests are renamed to ic_ledger_suite_state_machine_tests
    Review: Code changes matches release notes
  • c78e3549d Interface,Message Routing: Restore idle timeout functionality for XNetClientImpl (#2179)
    Notes: pool_timer(TokioTimer::new()) call has been added to Client::builder as the newer version requires it.
    Review: Code changes matches the release notes and look good.
  • 4b3c41b7c Interface,Networking: Remove unneed clones when serving requests from the grpc service (#2129)
    Notes: Changes has been made to bitcoin adapter grpc service to remove the use of clone and use iterator reference.
    Review: Code changes matche the release notes and look good.
  • 34b7182bf Interface,Networking(https-adapter): Bind the server socket before returning from start_server(). (#2150)
    Notes: Code changes has been made to make sure the socket server is binded before calling rt_handle_spawn
    Review: Code changed are okay and match the release notes.
  • b545f6b10 Interface,Networking: remove async mutexes in the BTC adapter (#2116)
    Notes: The use of tokio::sync:Mutex is replaced with std::sync::Mutex in the bitcoin adapter
    Review: Code changes matches the release nodes and look okay.
  • 6afff64b2 Owners(IDX): work around proc-macro2 non-determinism (#2176)
    Notes: Code changes apply patches = [“//bazel:rules_rust.patch”], a rules_rust patch to work around proc-macro2 non-determinism
    Review: Code changes matches the release notes and look goood.
  • 1e22810de Node: add a temporary measure to remove existing API BN certs (#2146)
    Notes: temporary fix to remove existing certificates of the API BNs for the switch from the HTTP- to the ALPN-ACME .
    Review: Code changes match the release notes and look okay.
  • 1eae28c1e Node(recovery): Set up empty recovery directory in guestOS (#2131)
    Notes: /var/lib/ic/data/recovery directory has been added with admin permission to simplify the recovery process.
    Review: Code changes match the release notes and look okay.
  • 2ce147d64 Node: allow read access to more hardware info for node_exporter (#2121)
    Notes: More hardware info like reading udev state data, proc pressure have been given access prometheus node_exporter.
    Review: Code changes match the release notes and look okay.

Chores:

  • cce8a1797 Consensus,Interface(recovery): Improve error messages (#2122)
    Notes: This code changes add a bit more detail to error message and improve formatting
    Review: Code changes match the release notes and look okay.
  • 0a5351777 Crypto,Interface: Upgrade core crates and use workspace version (#2111)
    Notes: A couple of packages has been upgraded to a newer version and are now using workspace version
    Review: Code changes match the release notes and look okay.
  • 5d1978e61 Execution,Interface,Message Routing: Drop CanisterQueue::QueueItem proto, part 2 (#1861)
    Notes: This is a follow-up to #1797. It drops the old CanisterQueue proto representation as now everything has been migrated to a newer version.
    Review: Code changes matches release notes and look okay.
  • 588ad7a46 Interface: upgrade rust version to 1.82 (#2137)
    Notes: The rust version has been upgraded to 1.82 with some little changes to support this upgrade.
    Review: Code changes matches release notes and look okay.
  • ee61d5074 Interface: Migrate all canisters to clap 4 (#2013)
    Notes: Upgrade all the usage of clap to version 4
    Review: Code changes matches release notes and look okay.
  • 947217661 Interface: upgrade and use workspace version for quote and proc-macro2, remove ring where not used (#2095)
    Notes: Minor package version upgrades, use of workspace version in proc-macro2 and the usage of ring has been removed.
    Review: Code changes matches release notes and look okay.
  • eadc89853 Interface,Networking: Upgrade opentelemetry crates (#2081)
    Notes: Opentelemetry has been upgraded from 0.23 to 0.26 and other packages are also upgraded to support it.
    Review Code changes matches release notes and look okay.
  • d3def2417 Owners: Upgrade hyper, rustls and reqwest (#2162)
    Notes: hyper, rustls and reqwest has been upgraded.
    Review: Code changes matches release notes and look okay.
  • dff73c67c Node: Fixup services (#1837)
    Notes: A cleanup has been done in services, and some other refactoring changes.
    Review: Code changes matches release notes and look okay.
  • 5ca5d3c5b Node: Update Base Image Refs [2024-10-17-0809] (#2093)
    Notes: Updates the base image references
    Review: Code changes matches release notes and look okay.

Refactoring:

  • ecc5be798 Interface,Networking(http-handler): Remove async handler for the v3 call module (#2100)
    Notes: async handler for V3 has been removed now as all the calls are made sync now.
    Review: Code changes match release notes and look okay

Other changes:

  • edde85e8c Execution,Interface,Networking: “feat(): Adjust max number of cached sandboxes” (#2184)
    Notes: DEFAULT_MAX_SANDBOX_COUNT is now increase from 1_000 to 2_000
    Review: Code changes match release notes and look okay.
  • 4e666d720 Interface,Networking: “feat(https-outcalls): Enable H/2 support for outcalls (#2142)” (#2159)
    Notes: This change reverts c831d22ab Interface,Networking(https-outcalls): Enable H/2 support for outcalls
    Review: Code changes match release notes and look okay

Proposal: 133799

Summary

  1. Build Hash: Build has from the proposal, local build and CDN matches and is “e76d04feeee6a05ee221d31f7c18ab2ac5247807a59469cd721fa326f78071f7”.
  2. Summary: The release notes matches the code changes
  3. Vote: I vote to adopt the proposals

Detailed Review:

8ac632c8e Interface,Networking, Consensus: Enable the hashes-in-blocks feature
Notes: HASHES_IN_BLOCKS_FEATURE_ENABLED has been enabled and made true.
Review: Code changes match release notes and look okay.

Proposal 133800

Summary

  1. Build Hash: Build has from the proposal, local build and CDN matches and is “188d36117771f3ddc211e615aee01f6affb7247dbc8067bf0760f717371f1c68”.
  2. Summary: The release notes matches the code changes
  3. Vote: I vote to adopt the proposals

Detailed Review:
I have verified that these code changes just revert a few code changes listed below

  • a6ef5933e Node: Revert “chore: Fixup services (#1837)”

  • f471ef26e Node: Revert “fix(): allow read access to more hardware info for node_exporter (#2121)”

  • 6027218e9 Node: Revert “feat: Upgrade GuestOS to 24.04 (#938)”

1 Like

Hey @dsharifi, can I ask if there were any blockers to supporting synchronous call handling on the NNS subnet prior to this release, or was the prior decision to exclude system subnets simply for the sake of testing this new feature on less critical subnets first?

Proposal 133798

Vote: ADOPT

Reason: Build is successful and hashes match, I’ve checked all commits in this release and they match their descriptions, therefore I’ve voted to adopt.

Reviews

Features:

[75dd48c38] Increased DEFAULT_MAX_SANDBOX_COUNT from 2k to 5k. Changed process eviction logic to take into account resident set size. EvictionCandidate and SandboxProcessStats structs have been extended with a new rss field, this is used to keep track of the memory usage of each process, these stats are updated for all processes every 10 seconds in a separate thread. evict method would previously only evict a fixed number of processes, now it keeps going until the total resident set size hasn’t been brought under a defined threshold. In get_sandbox_process eviction is not only triggered whenever the number of active processes in above the max_sandbox_count but also whenever the total resident size is above 50 GiB.
Finally 2 new helper methods have been added to process the total memory usage of all processes and update each process current memory usage.

[145c093b9] Already validated as part of previous releases, matches description.

[ec1cc659a] Already validated as part of previous releases, matches description.

[dbe869c88] Added on_low_wasm_memory hook function signature to the map of valid exported functions. TaskQueue has been modified to return OnLowWasmMemory execution task when the hook status is set to ready while peeking or popping the queue. is_empty and len methods have also been changed to account for the hook. Finally is_next_method_chosen method has been modified to early return if the task at the front of the task queue is a hook, the check is done by calling a new method is_hook, which currently only returns true for OnLowWasmMemory variant.

[944b8d0c3] Implemented Storable trait for Allowance<Tokens> so it can be persisted in stable memory.

[f55a7562c] Modified induct_message method to generate reject signals for certification versions above v19, while pushing both reject and accept signals for lower versions.

[c831d22ab] Added support to all http version instead of only http1 for outcalls.

[b1ffe8292] Modified enable_synchronous_call_handler_for_v3_endpoint to always return true, effectively enabling sync v3 endpoint on all subnets, including NNS one.

[0fadddd59] Moved mgmt_mac out of ICOSDevSettings into ICOSSettings and changed its type from an optional string to FormattedMacAddress. When initializing the config if the mac address cant be extracted from the deployment settings, now it is also attempted to get it by calling get_ipmi_mac.

[e31dc4c89] Implemented Storable trait for Account so it can be persisted in stable memory.

[00a3fb941] Moved the paths for external dependencies from WORKSPACE.bazel to a separate config.

Bugfixes:

[b7bf79f47] Modified canister execution metrics in execute_canisters_on_thread to ignore messages that ran 0 instructions and added a separate metric zero_instruction_messages, which is a counter that gets incremented whenever it happens.

[03dd6ee6d] Renamed ic-icrc1-ledger-sm-tests to ic-ledger-suite-state-machine-tests.

[c78e3549d] Preserve timeout functionality in XNetClient by setting pool_timer when building the client.

[4b3c41b7c] Modified get_successor_blocks and get_next_headers methods in the bitcoin adapter to use immutable references for BlockHash instead of owned values to avoid unnecessary cloning.

[34b7182bf] Moved instantiation of Http service and server binding outside rt_handle thread in start_server so that it happens before the method returns.

[b545f6b10] Replaced tokyo async mutexes in the BTC adapter with sync mutex from the standard library.

[6afff64b2] Added patch to rules_rust to fix non deterministic behaviour.

[1e22810de] Added temporary code to bootstrap-ic-node.sh to remove existing boundary nodes certificates.

[1eae28c1e] Modified setup-permissions.sh to create an empty directory with admin permissions to be used during subnet recovery.

[2ce147d64] Give prometheus node_exporter access to more hardware infos.

Chores:

[5d1978e61] Removed protobuf definitions and rust conversion code for deprecated canister queue item representation.

[cce8a1797] Improved error messages readability in rs/recovery.

[0a5351777] Bump version number of some packages and set them to use workspace version.

[588ad7a46] Bump rust version used to 1.82 and some minor code changes to satisfy clippy warnings.

[ee61d5074] Update system canisters to use clap 4.

[947217661] Bump version number for quote and proc-macro2 and set them to use workspace version. Removed ring dependency where it isn’t needed.

[eadc89853] Bump version number for opentelemetry crates.

[d3def2417] Bump version number for hyper, rustls and reqwest.

[dff73c67c] Changes seem to match the description.

[5ca5d3c5b] Updated Base Image Refs.

Refactoring:

[ecc5be798] Removed async call handler for the v3 endpoint, since all subnets support the sync version as of this release it is no longer needed.

Other changes:

[edde85e8c] Set DEFAULT_MAX_SANDBOX_COUNT to 2000.

[4e666d720] Revert c831d22ab.

Proposal 133799

Vote: ADOPT

Reason: Build is successful and hashes match, this release contains the same changes as the ones proposed with 133798, but has hashes in block feature enabled.

Reviews

[8ac632c8e] Set feature flag for hashes in block feature to true.

Proposal 133800

Vote: ADOPT

Reason: Build is successful and hashes match, I’ve checked all commits in this release and they match their descriptions, therefore I’ve voted to adopt.

Reviews

[a6ef5933e], [f471ef26e], [6027218e9] All commits revert changes introduced in previous releases.

2 Likes

The latest release candidate that will be proposed to the NNS subnet on Friday will enable the synchronous call endpoint next Monday. We went with a gradual rollout approach, where we rollout the feature to the NNS subnet last.

3 Likes