Response verification failed: Invalid tree root hash

Developers
1

Hi,

accessing our asset canister via the browser is producing an error Response verification failed: Invalid tree root hash. It was previously working fine but today it seems to only produce this error message.

When we attempt to redeploy our canisters we get more errors regarding ingress messages.

Building frontend...
69
Installing canisters...
70
Reinstalling code for canister gldt_swap_app, with canister ID ebrqv-vaaaa-aaaal-qb6zq-cai
71
Uploading assets to asset canister...
72
Fetching properties for all assets in the canister.
73
Starting batch.
74
Error: Failed while trying to deploy canisters.
75
Caused by: Failed while trying to deploy canisters.
76
  Failed while trying to install all canisters.
77
    Failed to install wasm module to canister 'x'.
78
      Failed to store assets in canister 'x'.
79
        Failed asset sync with canister ebrqv-vaaaa-aaaal-qb6zq-cai.
80
          Failed to create batch: The replica returned a rejection error: reject code SysTransient, reject message Ingress message 0xf42a19888b42c0c5b83e6cae75c9e62618bc7378731a563ee00be3aa7c134ed1 timed out waiting to start executing., error code None

81
Cleaning up project directory and file based variables
00:0182
ERROR: Job failed: exit code 1

Is it possible that the canister somehow trapped and reverted or that there is an issue with the subnet?

here is our subnet : https://dashboard.internetcomputer.org/subnet/e66qm-3cydn-nkf4i-ml4rb-4ro6o-srm5s-x5hwq-hnprz-3meqp-s7vks-5qe

dfx 0.19.0

1 Like
2

Hmm, it’s suddenly started working now. weird

3

Can we have a reason why it was failing ?
Some staging/production canisters was impacted, and since last weeks it’s not the first time we are facing this issue :confused:

2 Likes
4

What is the canister id of the canister that was failing?

When did the issue first occur and did you make any changes prior to the first occurrence? Such as upgrading DFX?

5

In that case, that was this canister : ebrqv-vaaaa-aaaal-qb6zq-cai
and we are using dfx 0.19.0 since the beginning for this project. Do you think dfx version can be the reason why we had this issue ? For some reason, just after we posted here it worked again, but yesterday for ~4hours it was impossible for us to deploy/access frontend application deployed on this canister.
This issue started to appear ~1week ago.

6

FYI it’s again down @NathanosDev

7

I cannot reproduce it right now, can you still?

If so, please check the network tab, for the failed request you will see a remote address, please grab that IP address and share it here. Also please get the Ic-Certificate response header and share that too.

8

It’s back online now.
We will try to update dfx and use that one.
On network tab, we have a lot of error 500 :confused: I’ll try to screen that next time we have it, if dfx upgrade is not solving our issue.

9

Im actually going through this currently. Is there any solution to this issue.

10

Hey @successaje, could you please try and provide as much of the information that I have requested above as possible? Thank you.

1 Like
11

Here is a somewhat detailed information in that issue -

12

This looks like it’s a problem of the subnet being too busy. Your subnet has been identified as one of the problematic ones in this post: Subnets with heavy compute load: what can you do now & next steps - Developers - Internet Computer Developer Forum (dfinity.org). Please refer to that post for advice on what to do and plans going forward.

1 Like
13

Thanks for the guidance