You can see the sha256 results for each build from each reviewer on the codegov portal. Each reviewer is required to post a screen capture of their artifacts.
My environment is windows 11 running wsl2 and Ubuntu 22.04 LSR (not the desktop version) and podman. I run the script that is posted in the proposal without modification. I have also been able to run this script in Ubuntu 22.04 desktop in a virtualbox virtual machine on the same pc, but I usually prefer wsl2 instead. I created the VM for these ic-os verifications back when docker was required. Hence docker is installed on the VM and directly on my pc, but I donāt intentionally use it unless it is being used by the proposal script.
I canāt quote the stats from memory, but I do believe that people have struggled to build the replica due to either lack of hard drive space, lack of memory, or lack of sufficient cores. If it were possible to reduce the hardware requirements then that may result in more people who can perform the work to build the replica.
I hope this helps. Thank you so much for offering feedback and guidance that can help improve our understanding and contributions to these reviews. I love seeing this type of collaboration. DFINITY is doing a great job of engaging the community and enabling independent reviews and voting. Itās very impressive and exciting.
My issue ended up being that I was running an out-of-date version of build-ic.sh. Once I switched to what is provided within the repo, my build completed fine.
There are three sets of hashes: setupos, hostos & guestos.
For guestos, a couple of reviewers had 16 GB RAM on which they attempted to build on WSL2. The build failed reporting "no enough space " or something similar.
1.1 i had the same issue. I modified the config to NOT mount ātmpfsā. Then the build itself succeeded; but produced incorrect hashs.
All reviewers with correctly configured machines were able to produced correct hashes for guestos.
None of the reviewers were able to produce correct hashes (and hashes were not consistent amongst reviewers) for setupos. For hostos , i verified that hashes were not consistent amongst reviewers (did not check for correctness)
Iāve been trying to get to the bottom of the build indeterminism. First, some further background on the issue:
If HostOS is not deterministic, SetupOS wonāt be either because SetupOS contains a HostOS image inside of it. So the source of the indeterminism should be isolated to HostOS.
@wpb and @Gekctek, you two were running very similar build environments and almost had the same builds. You had differing update-img-test images, and weāre looking into this, but this is a separate issue that we believe will be straightforward to resolve.
You both said the build environment you were running was:
Windows 11
WSL2
Ubuntu 22.04
Podman
Iām curious to know what Podman version you were running.
Anyone who ran the test, could you take a moment to upload your hostOS disk-img.tar.gz file to a shared folder: tinyurl .com/2p8wfuz9
NOTE: You are unable to share link in these threads, so please just remove the space in the URL.
Iāve created folders with each of your account handles. Please place the file in your respective folders. This way, I can inspect the issue and identify the source of the indeterminism. And after uploading your images, can you also comment in this thread your OS, Podman version, and any other build environment information you think might be relevant.
Additionally, @wpb and @Gekctek, can you both upload your update-img-test.tar.gz files so that I can take a look at that issue.
We believe we have found and resolved the source of the indeterminism!
For those curious about the issue:
In the bootloader, microcode updates were being added to the initrd based on the detected CPU of the machine building the image. We solved the issue by disabling initramfs from including microcode updates.
Note: the reason our existing testing infrastructure did not discover the issue is that our machines were all using the same AMD processors, which meant they were getting the same microcode updates, causing them to create deterministic images.
Iām not sure if this change will be included in the next update proposal, but if not, it should definitely be included in the subsequent one!
It should, yes! Because a SetupOS image contains GuestOS and HostOS images, we believe the SetupOS indeterminism was a result of the HostOS indeterminism.
From the looks of the builds in the codegov review for the latest replica update proposal, we still have a HostOS indeterminism issue. Apologies for the pre-mature celebration. Weāve either overlooked something, or this is a new issue.
Like before, NNS proposal 122529 is just a replica update proposal, and the indeterminism is just in HostOS and SetupOSānot GuestOS. So this proposal can still pass.
I am posting another link to a shared drive, and requesting that people upload their HostOS disk-img.tar.gz file.
Thanks again for all your help and patience. We will get this resolved!
