Non reproducibility of ic-os build

[andrewbattat]

Hi all!

My apologies for the long delay. We have been hard at work on other features and are now getting very close to rolling out our first-ever HostOS upgrade. Unfortunately, we are still dealing with some determinism issues.

As before, GuestOS and GuestOS update images are still deterministic. It is only in HostOS that we are encountering determinism issues (and, by extension, SetupOS).

Since my last post, we have solved some of the nondeterminism, but we are still chasing down more sources of nondeterminism. And, like before, we are still unable to reproduce the issue in our local testing, meaning HostOS is still deterministic for our tests (which is great), but this means we again need your help collecting HostOS images.

I’m reaching out to all the community members who verified the build and posted in this thread: DSCVR

@ZackDS, @wpb, @cyberowl, @massimoalbarello, @Gekctek, @tiago89, @ilbert, @Zane

I would greatly appreciate if all who are available would take the time to upload your HostOS update image builds (update-img.tar.gz) for this proposal to this shared folder

I have created a folder for each of you.

Please also include your build environment details (processor, memory size, operating system, anything else you think is applicable). You may post this information in this thread, message me directly, or include it as a text file in your folder.

Thank you all for helping to build and maintain the Internet Computer! I’m happy to answer any questions.

P.S.

@Zane and @cyberowl, you both produced the same images we did on our end, so we do not need your HostOS update images. I have just tagged you both for visibility.

@cyberowl, your update-img-test.tar.gz was nondeterministic, however. So if you could, we would greatly appreciate if you could upload that image.

[wpb]

I have uploaded my update-img.tar.gz file from my ~\ic\artifacts\icos\hostos directory as requested. I also verified that all specs for my environment are the same as my previous posts quoted below. Let me know if there is anything else you need to know about my build or my environment.

@wpb

My environment is windows 11 running wsl2 and Ubuntu 22.04 LSR (not the desktop version) and podman. I run the script that is posted in the proposal without modification.

@wpb

12th Gen Intel(R) Core™ i7-12700T 1.40 GHz
32.0 GB (31.7 GB usable)
64-bit operating system, x64-based processor

Windows 11 Pro Version 22H2
Ubuntu 22.04.2 LTS
podman version 3.4.4

[cyberowl]

Specs

The disk that I am using is older disk drive and not NVMe

podman 3.4.4

new verification from same proposal, since I deleted old one

uploaded

update-img tar and test file

[ZackDS]

Built it again and uploaded the image with the config specs. Not at home so used a different PC this time.

Ubuntu 22.04.3 LTS up to date on a i7-7700 with 32 GB ram and a SATA3 SSD, podman version 3.4.4

[Gekctek]

I cant quite figure out how to get the hostos with the new script /gitlab-ci/tools/repro-check.sh
It seems to create a tmp directory, which then gets deleted after the process. What am i missing?

##### HOSTOS SHA256SUMS #####
/tmp/tmp.QXmubIYVwL/ic/artifacts/icos/hostos /tmp/tmp.QXmubIYVwL/ic
ed25ccacb97bcfbceea5efe63566d133837dbc0b681778b1171f07e4b77bc48d *disk-img.tar.gz
a712183296a0ff2679d9cfe776113370afdb463ba1ab4de2d541f0e6405c2217 *disk-img.tar.zst
ccfc42662179a2a674296907f883b22c918aeacb562077de0c613303ff73523e *update-img-test.tar.gz
fc54fe5d9247d0ae039c3c06ce5e705e30f390847c852b8b2d1acaeedbbb4421 *update-img-test.tar.zst
93265614dd82b1ce24d83bcba88fe964237b9d6103c4d49cd4d70f1b02e3222a *update-img.tar.gz
3f4c53be9dbdbcfe7fd42d65462f309b7b9a8b2d6e051d48d81e90afb9fc7458 *update-img.tar.zst

[andrewbattat]

You should find your images outputted in the artifacts folder of the IC repo you built your images in.

ic/artifacts/icos/hostos/

[Gekctek]

Nah, nothing. I have searched my entire disk for the artifacts with no luck. Im currently trying to modify the script to not delete them

What script is everyone else running?
Im using gitlab-ci/tools/repro-check.sh

[tiago89]

@Gekctek I found it searching with “find” utility (ubuntu):

• find . -iname update-img.tar.gz

And it found it on bazel cache:

• ./.cache/bazel/_bazel_root/6d065581cce7ad9076e3b8db2b3afaf0/execroot/ic/bazel-out/k8-opt/bin/ic-os/hostos/envs/prod/update-img.tar.gz

Hope it helps.

[tiago89]

@andrewbattat have uploaded to shared folder my copy.

I am using an external (shared) server in a cloud:
OS: ubuntu 22.04
Disk: 160 GB
Memory: 16 GB
CPU: Intel Xeon Processor (Skylake, IBRS) - 64 bits - 4 core - 2GHz

Hope this helps,
Tiago

[wpb]

I use the script that is in the proposal. It’s just a copy paste because Dfinity fills in all the current proposal relevant information. My artifacts folder doesn’t get deleted at the end of the build.

[wpb]

Thank you @cyberowl @ZackDS @Gekctek and @tiago89 for following up on this request by @andrewbattat.

[Gekctek]

@tiago89

I found it searching with “find” utility (ubuntu):

• find . -iname update-img.tar.gz

And it found it on bazel cache:

• ./.cache/bazel/_bazel_root/6d065581cce7ad9076e3b8db2b3afaf0/execroot/ic/bazel-out/k8-opt/bin/ic-os/hostos/envs/prod/update-img.tar.gz

Hope it helps.

Thank you, that worked

@andrewbattat
File is uploading to shared folder

12th Gen Intel(R) Core™ i7-12700KF (20 CPUs), ~3.6GHz
32 GB RAM
Latest Windows 11
WSL2 Ubuntu 22.04
podman 3.4.4

[ilbert]

@andrewbattat I’ve uploaded the update image build to my folder.

My build machine is an AWS EC2 instance, with these details:

• OS: Ubuntu 22.04.2 LTS (x86_64)
• CPU: 8 vCPUs
• Disk: 110 GiB (General purpose 3 type)
• Memory: 32 GiB
• Podman: 3.4.4

I’ve also uploaded the SHA256SUMS file in my folder.

[Zane]

Not sure if It could help but this are the specs of the environment I build the replicas with:

[andrewbattat]

A quick update on IC-OS reproducibility:

We have solved most all of the HostOS/SetupOS nondeterminism issues. There are a few edge cases that we are aware of, but for the most part, all IC-OS images (GuestOS, HostOS, and SetupOS) should be deterministic.

The new repro-check script will check the reproducibility of all IC-OS images. The desired output should look like the following:

Please keep in mind that a replica upgrade = a GuestOS upgrade, so for replica upgrades, the only requirement is for the GuestOS to be deterministic. Of course, we want all images to be deterministic all the time, but a nondeterministic HostOS/SetupOS should not disqualify a replica release.

We will continue identifying and resolving any nondeterminism that comes up.

Thank you all for your work verifying IC releases!