Lost access to 2 out of 3 canisters of my project

I have a game running 3 canisters:

  • frontend: graci-aaaaa-aaaah-aaqjq-cai
  • backend: gwbe4-nyaaa-aaaah-aaqja-cai
  • high scores: ypg3d-6qaaa-aaaah-aasgq-cai

I am not the person who started the development, so I’m not the controller of any of the canisters, however my principal id has been added to the admins for all. I used to be able to upgrade all 3 canisters, now I can only upgrade frontend.

I used to update using dfx deploy --network ic for which I now receive:
The Replica returned an error: code 3, message: "Canister gwbe4-nyaaa-aaaah-aaqja-cai has no update method 'wallet_call'"

This can be fixed by adding the --no-wallet argument, which used to not be necessary, I imagine this is due to upgrading from dfx 0.8.1 to dfx 0.8.4.

Now I receive this error:
The replica returned an HTTP Error: Http Error: status 403 Forbidden, content type "", content: Requested canister rejected the message

My principal id locally is: g2h47-5pqqj-76vk6-plcc4-ub4wk-g646a-fzxgi-fxurm-tir3k-a2eij-pae which is among the admins for that canister and I used to be able to perform updates.

Canister in question: Principal gwbe4-nyaaa-aaaah-aaqja-cai | ic.rocks

Some ideas what may causes the issue / extra info

  • I saw someone had a similar issue, and their solution was to delete the .dfx folder, I tried that, not sure if that can cause harm. I still have the same principal id
  • There was an update from dfx 0.8.1 to 0.8.4, not sure if that did anything.
  • Number of cycles on canister should be more than enough for the update

Of course the main question is does anyone know why this may be happening and possible fixes for it?

But also, how come I can still update the frontend canister but not the two other ones? Shouldn’t it be all 3 or none of them?

Thank you for taking the time to read this, and any help is very much appreciated!

2 Likes

If you have access to the source code, is it possible the two canisters are explicitly rejecting the upgrade for some reason, e.g. by checking that the new installer is the same as the original installer?

1 Like

Motoko canisters will also reject upgrades if they have pending callbacks.

If that may be the case, you can ‘stop’ canisters using dfx to fail all pending calls and stop accepting new ones before attempting the upgrade. ‘dfx canister stop …’ or similar is your friend here.

1 Like

That makes sense! For a second I thought this would be it. But when I tried running
dfx canister --network ic --no-wallet stop --all
I’m getting:
The replica returned an HTTP Error: Http Error: status 403 Forbidden, content type "", content: Requested canister rejected the message

I assume this would rule out this option.

1 Like

I do have access to the source code, but I’m not quite sure what to look for. Is there an example you can point me to?

The canister seems to work just fine it seems, based on Principal gwbe4-nyaaa-aaaah-aaqja-cai | ic.rocks.

So my guess is that somehow your identity has changed, and you are not actually interacting as g2h47-5pqqj-76vk6-plcc4-ub4wk-g646a-fzxgi-fxurm-tir3k-a2eij-pae? What does dfx identity --network ic get-principal say?

Ah, reading your post some more, you say “so I’m not the controller of any of the canisters” but you are trying to upgrade the canister (dfx deploy). That doesn’t work, only a controller can install new code.

I am confused why rocks.ic doesn’t show the controllers or module hash for these canisters. Possibly a bug there?

But dfx can tell us this:

~/dfinity/dfx-in-nix $ dfx canister --no-wallet --network ic info gwbe4-nyaaa-aaaah-aaqja-cai
dfx.json not found, using default.
Controllers: ytmw6-niaaa-aaaai-qawvq-cai
Module hash: 0x5d5c318f49ca3ea3d40e5222768619653e8c32bdc28013b6ca005f59c22a919f
~/dfinity/dfx-in-nix $ dfx canister --no-wallet --network ic info ypg3d-6qaaa-aaaah-aasgq-cai
Controllers: ytmw6-niaaa-aaaai-qawvq-cai
Module hash: 0xf91e43e14cf70ba5668d7356b99c221e7faa1538606344cad9c2b4c73f3a8d0a
~/dfinity/dfx-in-nix $ dfx canister --no-wallet --network ic info graci-aaaaa-aaaah-aaqjq-cai
Controllers: ytmw6-niaaa-aaaai-qawvq-cai
Module hash: 0xe0df779f65fe44893d8991bef0f9af442bff019b79ec756eface2b58beec236f

so the controller of all of these is canister ytmw6-niaaa-aaaai-qawvq-cai. Here we get

~/dfinity/dfx-in-nix $ dfx canister --no-wallet --network ic info ytmw6-niaaa-aaaai-qawvq-cai
dfx.json not found, using default.
Controllers: o37zo-nk6dg-nyrho-ppa47-v4mhu-jidr5-lodme-ovpjt-c26rv-2kr75-4qe
Module hash: 0x1404b28b1c66491689b59e184a9de3c2be0dbdd75d952f29113b516742b7f898

so it is owned by o37zo-nk6dg-nyrho-ppa47-v4mhu-jidr5-lodme-ovpjt-c26rv-2kr75-4qe, and the code hash is that of cycles-wallet-0.7.2, according to ic.rocks (once it is up again).

2 Likes

He does state he was added as an admin though I’m not sure in what sense.

Perhaps you need to get dfx to use another wallet associated with one of the controllers.

On a mountain so can’t really help much more.

That seems to be the most obvious problem, since that’s what the error message says, however if I do type dfx identity --network ic get-principal I get g2h47-5pqqj-76vk6-plcc4-ub4wk-g646a-fzxgi-fxurm-tir3k-a2eij-pae

Here Principal gwbe4-nyaaa-aaaah-aaqja-cai | ic.rocks you will find a method called getAdmins that lists all principal ids who can make updates to the canister. This is how I’ve been able to make changes before.

No rush, it’s New Year’s Eve anyways, enjoy the day! Happy new year!

1 Like

Discusses the use of caller identification, if that’s any help

But if that’s the case, how was I able to install new code on the canisters until now? And how can I still update the graci-aaaaa-aaaah-aaqjq-cai canister?

I remember the original developer tried to add me as controller, however there can only be 1. And to make sure we don’t lose access to the canister we wanted multiple people to be able to update, so he added admins instead. You can see getAdmins() at https://ic.rocks/principal/graci-aaaaa-aaaah-aaqjq-cai

When you say update, do you mean upgrade (i.e. changing the code), or performing update calls? The admin list inside the canister can only affect application logic (update calls), and does not affect changing code (canister upgrade).

So are you sure you previously changed the code of these canisters (and not just data on the canister)? If so, then either you previously could use the wallet ytmw6-niaaa-aaaai-qawvq-cai, or someone changed the controllers.

BTW, multiple controllers is possible now.

1 Like

Sorry, wrong use of terminology. I meant upgrade throughout this thread, to change the code of a canister.

And yes, I’m pretty sure. Can be mistaken but I remember deploying 2 out of the 3 canisters multiple times. Maybe your BTW is the solution. We have lost contact with our original developer, but hopefully he would be able to add me as a controller and then all this won’t be a problem anymore.

Thanks a lot for all of your responses!

1 Like

What does this controller tree mean? Because the second principal id is the one I’m trying to upgrade with.

Oh and btw my friend is the controller however he can only use ic.rocks and the NNS since he’s using Windows where dfx didn’t work when we last tried it. Not sure if anything changed since.

1 Like

Oh, and it says (1 controller) and shows 2 ids. I’m confused.

One last thing, sorry for these spread-out questions. My friend has the principal id in the NNS which is the owner, but the NNS tells him, he’s not. Do you know why?


The canister gwbe4-nyaaa-aaaah-aaqja-cai has a single controller, namely ytmw6-niaaa-aaaai-qawvq-cai, a wallet canister. That canister has two controllers, namely o37zo-nk6dg-nyrho-ppa47-v4mhu-jidr5-lodme-ovpjt-c26rv-2kr75-4qe and g2h47-5pqqj-76vk6-plcc4-ub4wk-g646a-fzxgi-fxurm-tir3k-a2eij-pae.

So if you are g2h47-…, then you should be able to upgrade the canister gwbe4-… if you tell dfx to use that wallet. Try dfx deploy --wallet ytmw6-niaaa-aaaai-qawvq-cai.

3 Likes

Oh my god… IT WORKED! Thank you so much! There must have been an update in dfx. I used to never write --wallet. And now I just tried to deploy with --no-wallet which got rejected. I hope my mistakes will help others.

3 Likes