Discrepancy Between Local SHA-256 Wasm Hash and Canister Status's `module_hash`

[b3hr4d]

Hello Dfinity Community,

I am experiencing a discrepancy between the SHA-256 hash of a Wasm file calculated locally (both in JavaScript and Rust) and the module_hash reported in a canister’s status on the Dfinity platform. I’m hoping to gain insights or suggestions on why this might be occurring.

The Issue:

• When calculating the SHA-256 hash of a Wasm file locally using standard libraries (crypto.subtle.digest in JavaScript and ic_crypto_sha2::Sha256::hash in Rust), I consistently get the following hash:

  [
    55, 110, 3, 129, 195, 175, 128, 32,
    98, 31, 15, 105, 218, 1, 41, 12,
    116, 227, 94, 141, 195, 247, 62, 10,
    27, 44, 15, 217, 195, 143, 170, 83
  ]
  

• However, the module_hash obtained from the canister’s status is different:

  {173; 232; 148; 57; 254; 212; 152; 86; 41; 253; 22; 139; 49; 151; 110; 63; 40; 117; 127; 37; 152; 67; 159; 19; 39; 62; 152; 96; 15; 9; 225; 229}
  

Steps Taken:

1. Local Hash Calculation: Implemented a function in both JavaScript and Rust to calculate the SHA-256 hash of the Wasm file.
2. File Integrity: Ensured that the Wasm file is identical in all tests and is read in binary mode.
3. Hash Function Consistency: Used standard SHA-256 implementations in both programming languages.

Questions:

1. Preprocessing of Wasm File: Does Dfinity perform any preprocessing or modification on the Wasm file before calculating the module_hash?
2. Hashing Process: Is there a specific detail in the hashing process on Dfinity’s end that might differ from the standard SHA-256 implementation?
3. Environmental Factors: Could environmental factors influence the module_hash reported by the canister’s status?
4. Additional Insights: Are there known issues or considerations that might explain this discrepancy?

I would greatly appreciate any insights or guidance to help understand and resolve this inconsistency. Thank you for your time and assistance!

Best regards,
Behrad

[timo]

Could one of the two be gzipped and the other not?

@b3hr4d

Hashing Process: Is there a specific detail in the hashing process on Dfinity’s end that might differ from the standard SHA-256 implementation?

It is standard. A command line like sha256sum canister.wasm has always given me the same hash as dfx canister info <canister id>.

[b3hr4d]

I have also stored the wasm bytes inside the canister, generated the hash, and then installed it using the management canister, resulting different hash!

[timo]

@b3hr4d

resulting different hash!

You mean a third hash or one of the previous two?

[b3hr4d]

Hi Timo,

Thank you for seeking further clarification. Here’s the detailed scenario:

1. Local Hash Generation: The hash we generate locally using JavaScript from the Wasm file is consistent and matches our expectations.

2. Command Line Tool Confirmation: Additionally, using the sha256sum wasmfile.wasm command line tool yields the same result as the JavaScript-generated hash. This adds another layer of confirmation to our local hash calculation process.

3. Pre-Installation Hash: We also generate a hash from the Wasm bytes before installing the canister. This hash is identical to both the local JavaScript-generated hash and the sha256sum result, confirming the accuracy of our local process.

4. Post-Installation Hash Discrepancy: However, the module_hash obtained from the canister’s status after the installation (via the Dfinity platform) is different from these two identical hashes.

This indicates that the local hashes (both pre-installation, JavaScript-generated, and sha256sum confirmed) are consistent with each other. Yet, the hash changes in some way during or after the installation process on the Dfinity platform. The key question now is what might be causing this change in the hash post-installation. Is there a transformation, compression, or any other process that the Dfinity platform applies to the Wasm file during the installation that could alter its hash?

I appreciate your assistance and any further insights you or the community might have on this matter.

[timo]

How large is your wasmfile.wasm? Just trying to get a sense if it might get compressed in the process.

Does the module_hash obtained from the canister’s status actually change when you upgrade the canister?

Are you sure you are querying the right network with dfx canister status? I mean local vs mainnet with --network ic? (Sorry for asking this but this kind of stuff happens to me all the time.)

[Severin]

Can you tell us a bit more about how you install your canister? If you use dfx canister install --wasm <my wasm> no transformation should be applied. If you use dfx deploy and give a path tho a wasm in dfx.json, then dfx will do some modifications like gzipping or adding the candid interface in a custom section.

@b3hr4d

Yet, the hash changes in some way during or after the installation process on the Dfinity platform.

The IC itself does not transform the uploaded code before calculating the hash. After calculating the hash, it will add instrumentation to e.g. count how much compute was used so it knows how many cycles to charge. Any changes to the hash you can observe would happen on client (here probably dfx) side

[b3hr4d]

I’ve resolved the issue with the module_hash and wanted to share the outcome. It turns out I had two versions of my Wasm file: one with a Candid interface and one without. This difference was causing the varied module_hash values.

After sorting out the files and retesting, all methods (dfx deploy, dfx canister install --wasm <my wasm>, and local JavaScript and Rust hashing) are now giving the same module_hash. I’m attaching screenshots to show these consistent results.

Using dfx

Using another canister

And using wasm file

However, I still couldn’t pinpoint the exact cause of why the inter-canister installation was returning a different wasm_module hash at one point. Restarting everything seemed to solve that problem as well.

Special thanks to Severin and Timo. Your insights helped me identify my mistake, and now everything’s functioning correctly.

Best,
Behrad

[bdemann]

Hey Severin,

Could you clarify what modifications dfx makes during the deploy process? We’re conducting tests with Azle to verify that both the init and post-upgrade functions trigger as expected. However, we’ve encountered a problem where even if the Wasm binary hasn’t changed, running dfx deploy sometimes still upgrades the canister.

Here’s what we’re seeing:

• The hash of our Wasm binary: 0xba9bca82427a85aad7418caa2ba874c2132f9a59042dab5749c8b6c75d128e1f
• When I install the binary directly using dfx canister install --wasm <my_wasm>, as you recommended, the module hash matches the Wasm hash exactly.
• However, when using dfx deploy, I get inconsistent module hashes. Specifically, I end up with one of two different hashes:
  • 0x4e366dbf2e7ba7659bcd6f40d7f124d503367b3cb6b77709f584e96b8a4bc330
  • 0xd02d1111555cebf016cba17b3fcc8a848e94385934fdb790f1182912b8600a14

While I understand that the module hash might differ from the Wasm hash due to potential modifications by dfx, I’m concerned that the hashes generated by dfx deploy are inconsistent with each other. I was hoping that dfx would apply deterministic changes, resulting in the same hash whenever the binary is unchanged, and consistent behavior when dfx chooses to skip an installation or trigger an upgrade. Without that I’m concerned about reproducibility for our end users and our tests.

Do you have any insights into why these inconsistencies occur and whether dfx is making non-deterministic changes to the binaries?

Thanks!

[Severin]

Can you help me track down things a bit more? You can inspect the end result that dfx produces in .dfx/<network>/<canister name>.

@lwshang, we had significant changes in ic-wasm recently. Any chance that we have some non-deterministic ordering in there now? Maybe the custom sections?

Other places where non-determinism can show up:

• Rust compilation (which Azle uses behind the scenes) is notoriously non-deterministic, although if you do the same compilation on the same machine in the same directory then it’s deterministic in my experience. To double-check that, inspect your target folder and see if the wasm in there is always the same.
• gzip: If you don’t use -n, file names and timestamps are included. dfx does that properly, but if you do custom magic maybe you missed it somewhere?

If you can, it would be great if you can analyze the two binaries in the hopes of finding a difference. For metadata, I’d suggest using ic-wasm

[bdemann]

Inspecting .dfx/local/canisters/canister/canister.wasm

I checked the hash of the WASM file in
.dfx/local/canisters/canister/canister.wasm, and it matches the module hash
after installation via dfx deploy

ic-wasm

When I run

$ ic-wasm .dfx/local/canisters/canister/canister.wasm metadata

I get the following metadata:

icp:public candid:service
icp:public cdk:name
icp:public dfx

When I run the same command on the WASM file generated by Azle:

ic-wasm .azle/canister/canister.wasm metadata

It returns no meta data

When I drill down in the specific meta data content I get this
For the module hash: 0x4e366dbf2e7ba7659bcd6f40d7f124d503367b3cb6b77709f584e96b8a4bc330

ic-wasm .dfx/local/canisters/canister/canister.wasm metadata candid:service
service : {
  getAzleInitCalled : () -> (bool) query;
  getAzlePostUpgradeCalled : () -> (bool) query;
  getInitCalled : () -> (bool) query;
  getPostUpgradeCalled : () -> (bool) query;
}

$ ic-wasm .dfx/local/canisters/canister/canister.wasm metadata cdk:name
azle

$ ic-wasm .dfx/local/canisters/canister/canister.wasm metadata dfx
{
  "tech_stack": {
    "cdk": {
      "azle": {}
    },
    "language": {
      "javascript": {},
      "typescript": {}
    }
  }
}

For the module hash: 0xd02d1111555cebf016cba17b3fcc8a848e94385934fdb790f1182912b8600a14
I get the same results with this difference:

$ ic-wasm .dfx/local/canisters/canister/canister.wasm metadata dfx
{
  "tech_stack": {
    "cdk": {
      "azle": {}
    },
    "language": {
      "typescript": {},
      "javascript": {}
    }
  }
}

In the language object typescript comes before javascript

Could that small difference be the source of the change in hash?

Wasm generated from azle

I’ve verified that Azle is producing the same WASM file consistently (at least
from the same machine). Hashing the .azle/canister/canister.wasm file gives me
the following hash:

0xba9bca82427a85aad7418caa2ba874c2132f9a59042dab5749c8b6c75d128e1f

I can install this using
dfx canister install --wasm ./.azle/canister/canister.wasm canister
The module hash produced after installation matches the above hash.

I have disabled gzipping and we get we are still getting the same effect.

[Severin]

@bdemann

In the language object typescript comes before javascript

Could that small difference be the source of the change in hash?

Yes, that is the error. Any single-bit difference will (and should!) cause a change in hash. This is a bug in dfx and I don’t have a good workaround for you at the moment.

Note to self: internal ticket