Discrepancy Between Local SHA-256 Wasm Hash and Canister Status's `module_hash`

[b3hr4d]

Hello Dfinity Community,

I am experiencing a discrepancy between the SHA-256 hash of a Wasm file calculated locally (both in JavaScript and Rust) and the module_hash reported in a canister’s status on the Dfinity platform. I’m hoping to gain insights or suggestions on why this might be occurring.

The Issue:

• When calculating the SHA-256 hash of a Wasm file locally using standard libraries (crypto.subtle.digest in JavaScript and ic_crypto_sha2::Sha256::hash in Rust), I consistently get the following hash:

  [
    55, 110, 3, 129, 195, 175, 128, 32,
    98, 31, 15, 105, 218, 1, 41, 12,
    116, 227, 94, 141, 195, 247, 62, 10,
    27, 44, 15, 217, 195, 143, 170, 83
  ]
  

• However, the module_hash obtained from the canister’s status is different:

  {173; 232; 148; 57; 254; 212; 152; 86; 41; 253; 22; 139; 49; 151; 110; 63; 40; 117; 127; 37; 152; 67; 159; 19; 39; 62; 152; 96; 15; 9; 225; 229}
  

Steps Taken:

1. Local Hash Calculation: Implemented a function in both JavaScript and Rust to calculate the SHA-256 hash of the Wasm file.
2. File Integrity: Ensured that the Wasm file is identical in all tests and is read in binary mode.
3. Hash Function Consistency: Used standard SHA-256 implementations in both programming languages.

Questions:

1. Preprocessing of Wasm File: Does Dfinity perform any preprocessing or modification on the Wasm file before calculating the module_hash?
2. Hashing Process: Is there a specific detail in the hashing process on Dfinity’s end that might differ from the standard SHA-256 implementation?
3. Environmental Factors: Could environmental factors influence the module_hash reported by the canister’s status?
4. Additional Insights: Are there known issues or considerations that might explain this discrepancy?

I would greatly appreciate any insights or guidance to help understand and resolve this inconsistency. Thank you for your time and assistance!

Best regards,
Behrad

[timo]

Could one of the two be gzipped and the other not?

@b3hr4d

Hashing Process: Is there a specific detail in the hashing process on Dfinity’s end that might differ from the standard SHA-256 implementation?

It is standard. A command line like sha256sum canister.wasm has always given me the same hash as dfx canister info <canister id>.

[b3hr4d]

I have also stored the wasm bytes inside the canister, generated the hash, and then installed it using the management canister, resulting different hash!

[timo]

@b3hr4d

resulting different hash!

You mean a third hash or one of the previous two?

[b3hr4d]

Hi Timo,

Thank you for seeking further clarification. Here’s the detailed scenario:

1. Local Hash Generation: The hash we generate locally using JavaScript from the Wasm file is consistent and matches our expectations.

2. Command Line Tool Confirmation: Additionally, using the sha256sum wasmfile.wasm command line tool yields the same result as the JavaScript-generated hash. This adds another layer of confirmation to our local hash calculation process.

3. Pre-Installation Hash: We also generate a hash from the Wasm bytes before installing the canister. This hash is identical to both the local JavaScript-generated hash and the sha256sum result, confirming the accuracy of our local process.

4. Post-Installation Hash Discrepancy: However, the module_hash obtained from the canister’s status after the installation (via the Dfinity platform) is different from these two identical hashes.

This indicates that the local hashes (both pre-installation, JavaScript-generated, and sha256sum confirmed) are consistent with each other. Yet, the hash changes in some way during or after the installation process on the Dfinity platform. The key question now is what might be causing this change in the hash post-installation. Is there a transformation, compression, or any other process that the Dfinity platform applies to the Wasm file during the installation that could alter its hash?

I appreciate your assistance and any further insights you or the community might have on this matter.

[timo]

How large is your wasmfile.wasm? Just trying to get a sense if it might get compressed in the process.

Does the module_hash obtained from the canister’s status actually change when you upgrade the canister?

Are you sure you are querying the right network with dfx canister status? I mean local vs mainnet with --network ic? (Sorry for asking this but this kind of stuff happens to me all the time.)

[Severin]

Can you tell us a bit more about how you install your canister? If you use dfx canister install --wasm <my wasm> no transformation should be applied. If you use dfx deploy and give a path tho a wasm in dfx.json, then dfx will do some modifications like gzipping or adding the candid interface in a custom section.

@b3hr4d

Yet, the hash changes in some way during or after the installation process on the Dfinity platform.

The IC itself does not transform the uploaded code before calculating the hash. After calculating the hash, it will add instrumentation to e.g. count how much compute was used so it knows how many cycles to charge. Any changes to the hash you can observe would happen on client (here probably dfx) side

[b3hr4d]

I’ve resolved the issue with the module_hash and wanted to share the outcome. It turns out I had two versions of my Wasm file: one with a Candid interface and one without. This difference was causing the varied module_hash values.

After sorting out the files and retesting, all methods (dfx deploy, dfx canister install --wasm <my wasm>, and local JavaScript and Rust hashing) are now giving the same module_hash. I’m attaching screenshots to show these consistent results.

Using dfx

Using another canister

And using wasm file

However, I still couldn’t pinpoint the exact cause of why the inter-canister installation was returning a different wasm_module hash at one point. Restarting everything seemed to solve that problem as well.

Special thanks to Severin and Timo. Your insights helped me identify my mistake, and now everything’s functioning correctly.

Best,
Behrad