Broken Replica Source Code Verification

[levi]

Hello,
Is there a correct way to verify the source code of a blessed replica proposal now that the binaries are using compression?

For proposal 54964 , the git commit is: 0ef2aebde4ff735a1a93efa342dcf966b6df5061 , and the release_package_sha256_hex hash specified in the proposal is

.
When I checkout the ic repo at the commit: 0ef2aebde4ff735a1a93efa342dcf966b6df5061 and build the code with the build commands in the readme, the build is successful but the hashes are different than the one in the proposal

@diegop
@roman-kashitsyn
How can I verify the source code of the replica?

[roman-kashitsyn]

The canister module GZip compression that I implemented recently is not related to this issue.

When I try to build the IC OS image from the same commit, I get yet another hash:

$ git status
HEAD detached at 0ef2aebd

$ ./gitlab-ci/tools/docker-run ./gitlab-ci/tools/build-ic

IC-OS Image
05cd757019e276af68a7e2e178dec73ae095131af841d0b44a9c03c947c2d399  update-img.tar.gz

There might be an issue with the build reproducibility; I asked our release engineers for clarification.

[levi]

@roman-kashitsyn

There might be an issue with the build reproducibility; I asked our release engineers for clarification.

Thanks, If I remember correct it is the extracted root.img file that is different.

[levi]

I’m seeing the same thing when building for proposal 56257
the proposal hash is

at commit 3ad313dcda03e2db45e81d02c5f931fdf3bf5bc1

When I checkout 3ad313dcda03e2db45e81d02c5f931fdf3bf5bc1 and build on my machine:

[sat]

@levi do you still see issues with reproducibility?

[levi]

@sat 2e16bb didn’t match, 3d6fc11 matched, 07954f didn’t match, dcb2d23 matched, and c273e3a matched.

[levi]

@sat 60222 commit: e7b57fc9
$ git status
HEAD detached at e7b57fc9 nothing to commit, working tree clean

[sat]

@levi reproducibility is hard…
Our IDX team made some changes recently, so that might be why the last few releases worked better.
Would be awesome if you could continue checking in the future and ping me (DM is great) or respond to the announcement message if you get a mismatch.

For each release our team does multiple (say 5 or so) fully independent builds to verify reproducibility. But it’s still not a proof that there won’t be a mismatch in some case or on some system.

[levi]

Sure, I’ll post on the new replica threads if there’s a mismatch .

[jzxchiang]

Thanks for doing the hard (but critical) work of replica verification!

[northman]

While performing replica verification the hashes do not match.

I think I am meeting the verification prerequites as I am running:

Ubuntu/WSL2Ubuntu 22.04.2 LTS
Podman version 3.4.4
git version 2.34.1

I have attempted to verify the most receive replica changes and both times the hashes do not match.

I believe it is a toolchain issue and not a code integrity issue.

Any suggestions? Is there any other things that need to be on a particular version?

[wpb]

Here is the most recent forum post about replica release e4843a1. I ran into the same issue you described, but the issue was that the script fails the wget command when it tries to download the latest update-img.tar.gz file because the file already exists in my ic folder from the build last week. I deleted the file from my ic directory and then reran the build using the script in the proposal and then both sha256 sums match the payload.

Your verification prerequisites are correct…ubuntu 22.04, wsl2, podman, and git

BTW, if you like to perform these verifications, then you should check out the CodeGov and the Voting Challenge portals on DSCVR. You might find them to have interesting information.

[northman]

So I am still unsuccessful in compiling the replica code.
I have removed the ic,.bazel and .bazelisk directories.

I am getting an error when running the verification script.

Writing manifest to image destination
Storing signatures
Error in atexit._run_exitfuncs:
Traceback (most recent call last):
File “toolchains/sysimage/docker_tar.py”, line 260, in
atexit.register(lambda: os.remove(tar_name))
PermissionError: [Errno 1] Operation not permitted: ‘/tmp/tmp55m02r00.tar’

Any particular reasons why the script would fail?

[northman]

I built on a different PC with Ubuntu 2204 and podman and the used the NNS validation script posted with the proposal. It built cleanly and the hash matched. I will now try to find what is off the baseline.

[bjoern]

I think that I am observing a similar pattern. I have two builders, one in an Ubuntu 22.04 Virtual Box VM and one in an Ubuntu 22.04 KVM. Both use exactly the same setup – same version of podman, docker.io, …, everything. Yet the build succeeds on one and fails on the other.

For some time I thought that one VM had more resources than the other, but that first doesn’t seem to change things and second I can observe disk and memory use and it never gets to even 50% during the build.

@northman what’s the size of the image file with the mismatching hash? I get some 22 MB or so.

[northman]

So it depends… currently it is only about 2M. I think the issue pops up when it is not compiled on a platform meeting the published the spec. WSL2 makes available only half the size of the physical ram and the swap is too low. I am uncertain why we need 8 cores but I can confirm that it works fine on the baseline defined by Dfinity 8 cores/16G, 100G swap).

[northman]

I ran the icos build verification script on the same computer but ran it on bare metal vs inside of WSL2.
The computer is only an I5 quad core with 8 Gig of RAM vs the 8 core 16G per the specs.
It compiled flawlessly and the hashes matched.
Under WSL2 it failed.

I think I will try bumping the memory so it will run on the WSL2 side.

I do not think it is an issue with the icos build script, rather it is a resource issue on the PC.

[bjoern]

I don’t think the number of CPU cores matters. I ran it in a VM with a single CPU core. Build takes hours, but completes successfully. Memory and disk space may matter but I couldn’t see a clear pattern for now.