Voting is open for a new IC release - b90edb98

Hello there!

we are happy to announce that voting is now open for a new IC release.
The NNS proposal is here: IC NNS Proposal 61004.

There are lots of exciting new features and bugfixes. Here is a more complete list of changes:

* Consensus: Fix to avoid purging initial dealings
* Consensus: Populate ECDSA summary in genesis CUP
* Consensus: Remove LMDB version test
* Crypto: Remove ThresholdSigVerifier<CatchUpContent> from CryptoComponentForVerificationOnly
* Crypto: Adapt CSP-APIs to propagate potential RPC errors
* Crypto: Adjusting config of ic-crypto-csp binary
* Crypto: Test parallelization of CSP vault calls
* Execution: Add bitcoin_get_utxos endpoint to management canister
* Execution: Add idle consumption rate to canister status
* Execution: Add the `bitcoin_get_balance` endpoint to the management canister
* Execution: Bitcoin: Support for syncing bitcoin mainnet
* Execution: Restore instruction limit for install_code messages on verified subnets
* Execution: Use new heartbeat code
* Message Routing: read_state: allow fetching full routing table
* Message Routing: Bump `CURRENT_CERTIFICATION_VERSION` to `V9`
* Message Routing: Speed up `CanisterQueues::has_input()`
* Message Routing: Speed up `induct_messages_on_same_subnet()`
* Message Routing: Speed up handling of untouched canisters
* Networking: Canister http client setup
* Networking: Explicitly pass the tokio runtime instead of assuming the call context is within it
* Networking: Make regedit main async
* Networking: Make the Response type of NonBlockingChannel an associated type
* Networking: Move the LocalStore in its own crate
* Node: Fix SELinux context of /etc/machine-id on first boot
* Node: Install 'zstd' package to support archives with Zstandard (zstd) compression
* Node: Policy issues for manageboot
* Runtime: Limit ECDSA signature queue size
* Runtime: Move xnet msg checks to routing
* Various bug fixes and test updates

And a complete list of changes can of course be found on github.
Please reply to this message if you have any questions or comments.

10 Likes

I seem to see integration with Bitcoin.

3 Likes

Yes last few releases have been adding more and more features related to Bitcoin integration

6 Likes
  • Execution: Add idle consumption rate to canister status
  • Execution: Use new heartbeat code

These sounds really interesting. Any chance you can explain what they do? I’m not sure I understand what “idle consumption rate” and “new heartbeat code” refer to.

  • Networking: Canister http client setup

Is this for the feature that allows canisters to make external HTTP requests?

1 Like

local build mismatch

2 Likes

Yes. This is related to that feature that allows canisters to make outgoing HTTP requests. But there is more work that needs to come into the feature before it becomes generally available.

1 Like

@levi Thanks so much for reporting a mismatch. Let’s try to get to the bottom of it.
We tested it on 4 independent machines at DFINITY for this release, but I’m now retrying on a completely new (clean) machine that I personally own, at my home.

Can you please try to rebuild with these commands, to avoid possible issues with some stale data:

docker image prune -a
docker builder prune
git fetch
git checkout b90edb9897718730f65e92eb4ff6057b1b25f766
git clean -ffdx
./gitlab-ci/tools/docker-build-ic

UPDATE: local rebuilding just finished on one of my personal (home) machines with the expected checksums:

IC-OS Image
1a5b1477f0c9001f82623f81679fbe80f773d3ae76c50b95f04f58b367c384e5 *update-img-test.tar.gz
50b59237df70d50b77af06b8c6737585cb5490b276a81d3797e6310a11c6fbd1 *update-img.tar.gz

The machines in question run Ubuntu 20.04 with the latest updates.

$ cat /etc/lsb-release
DISTRIB_ID=Ubuntu
DISTRIB_RELEASE=20.04
DISTRIB_CODENAME=focal
DISTRIB_DESCRIPTION="Ubuntu 20.04.4 LTS"
$ uname --all
Linux satbox 5.4.0-96-generic #109-Ubuntu SMP Wed Jan 12 16:49:16 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux

It would help if you could provide similar info from your side.

3 Likes

These sounds really interesting. Any chance you can explain what they do? I’m not sure I understand what “idle consumption rate” and “new heartbeat code” refer to.

  • Execution: Add idle consumption rate to canister status

The idle consumption rate in this commit refers to the idle rate consumption of resources that the canister is using, and it could be used to compute the freezing threshold in cycles.

  • Execution: Use new heartbeat code

The second change refers to a refactoring needed for deterministic time slicing, which is a work in progress.

6 Likes

Thanks!

$ cat /etc/os-release
NAME="Fedora Linux"
VERSION="36 (Workstation Edition)"
ID=fedora
VERSION_ID=36
VERSION_CODENAME=""
PLATFORM_ID="platform:f36"
PRETTY_NAME="Fedora Linux 36 (Workstation Edition)"
ANSI_COLOR="0;38;2;60;110;180"
LOGO=fedora-logo-icon
CPE_NAME="cpe:/o:fedoraproject:fedora:36"
HOME_URL="https://fedoraproject.org/"
DOCUMENTATION_URL="https://docs.fedoraproject.org/en-US/fedora/f36/system-administrators-guide/"
SUPPORT_URL="https://ask.fedoraproject.org/"
BUG_REPORT_URL="https://bugzilla.redhat.com/"
REDHAT_BUGZILLA_PRODUCT="Fedora"
REDHAT_BUGZILLA_PRODUCT_VERSION=36
REDHAT_SUPPORT_PRODUCT="Fedora"
REDHAT_SUPPORT_PRODUCT_VERSION=36
PRIVACY_POLICY_URL="https://fedoraproject.org/wiki/Legal:PrivacyPolicy"
VARIANT="Workstation Edition"
VARIANT_ID=workstation
$ uname --all
Linux fedora 5.17.9-300.fc36.x86_64 #1 SMP PREEMPT Wed May 18 15:08:23 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux

I run these commands:

docker image prune -a
docker builder prune
git fetch
git checkout b90edb9897718730f65e92eb4ff6057b1b25f766
git clean -ffdx
./gitlab-ci/tools/docker-build-ic

I sent you a message with the build logs.

@levi, thank you for providing this information. The differences on Fedora happen due to Btrfs filesystem. I’ve compared the image I get from Fedora 36 with default configuration and I got the same SHA cryptographic digest as you and the differences seen on the images showed that the directories have different hard link count. Btrfs is differnt with that by design.

We will take note of this and improve our utility script. Until then, please verify the build on Ubuntu 20.04 machine. You can do so using one of the public cloud providers and feeding the following cloud-config:

#cloud-config
apt:
  sources:
    docker.list:
      source: deb [arch=amd64] https://download.docker.com/linux/ubuntu $RELEASE stable
      keyid: 9DC858229FC7DD38854AE2D88D81803C0EBFCD88

packages:
  - sudo
  - git
  - apt-transport-https
  - ca-certificates
  - curl
  - gnupg-agent
  - software-properties-common
  - docker-ce

system_info:
  default_user:
    name: ubuntu
    sudo: ALL=(ALL) NOPASSWD:ALL
    primary_group: ubuntu
    groups: docker
    shell: /bin/bash

runcmd:
  - sudo -u ubuntu git clone https://github.com/dfinity/ic.git /home/ubuntu/ic
  - sudo -u ubuntu bash -c "cd ~/ic && git fetch && git checkout b90edb9897718730f65e92eb4ff6057b1b25f766"
  - sudo -u ubuntu bash -c "cd ~/ic && CI_DISPOSABLE_ENVIRONMENT=true ./gitlab-ci/tools/docker-build-ic"

If you save that under /tmp/cloud-config.yaml, you can make the build on AWS with the following command:

aws ec2 run-instances --image-id ami-02584c1c9d05efa69 --region eu-central-1 --count 1 --instance-type c3.4xlarge --key-name <key> --block-device-mappings 'DeviceName=/dev/sda1,Ebs={VolumeSize=100}' --user-data file:///tmp/cloud-config.yaml

AMI is official Ubuntu image.

Thank you again for providing this information, we will improve our validation utility script and instructions so this doesn’t happen again. Please let me know if this doesn’t clear your doubts or concerns or if you have any questions about it.

2 Likes

@marko thanks for looking into this. I’ll try to verify the build on ubuntu using a bootable usb on my machine.
I’ll keep an eye out for the build to match on Fedora.

Hi @levi. I’m sharing the snippet with which you can see I got the same SHA digest on Fedora 36 as you did and with which you can see the exact differences that happen due to Btrfs.

$ mkdir /tmp/b90edb9897718730f65e92eb4ff6057b1b25f766
$ cd /tmp/b90edb9897718730f65e92eb4ff6057b1b25f766
$ mkdir ubuntu fedora
$
$ curl -sS https://download.dfinity.systems/ic/b90edb9897718730f65e92eb4ff6057b1b25f766/guest-os/update-img/update-img.tar.gz -o ubuntu/update-img.tar.gz
$ curl -sS https://download.dfinity.systems/ic/fedora36/b90edb9897718730f65e92eb4ff6057b1b25f766/guest-os/update-img/update-img.tar.gz -o fedora/update-img.tar.gz
$ sha256sum */update-img.tar.gz
303926f3d814c084dd92e8c2f26a2513c26f2395796dce4187b59c6da185df20  fedora/update-img.tar.gz
50b59237df70d50b77af06b8c6737585cb5490b276a81d3797e6310a11c6fbd1  ubuntu/update-img.tar.gz
$
$ curl -sS https://download.dfinity.systems/ic/fedora36/b90edb9897718730f65e92eb4ff6057b1b25f766/guest-os/update-img/boot.diff -O
$ curl -sS https://download.dfinity.systems/ic/fedora36/b90edb9897718730f65e92eb4ff6057b1b25f766/guest-os/update-img/root.diff -O
$ # you can inspect the diff that we can see with diffoscope (upper diff is produced by commands below)
$ (cd fedora; tar -xzf update-img.tar.gz)
$ (cd ubuntu; tar -xzf update-img.tar.gz)
$ sudo diffoscope ubuntu/boot.img fedora/boot.img
$ sudo diffoscope ubuntu/root.img fedora/root.img
1 Like