Internal Server Error - IC is down?

Developers
1

I cannot access my assets canister since a couple of minutes. Likewise, cannot access the Motoko Playground nor the internet identity.

I either get 500 errors Internal Server Error or invalid SSL certificate.

https://status.internetcomputer.org/ reports no errors :thinking:.

Anyone notice the issue too?

2

Yes, even https://nns.ic0.app/ cannot be opened

3

Everything seems to be down :open_mouth:

4

Good to hear I am not alone.

The dashboard also displays that everything is alright :man_shrugging:

6

I’ve used this in the past for dynamic ssl allocation. GitHub - auto-ssl/lua-resty-auto-ssl: On the fly (and free) SSL registration and renewal inside OpenResty/nginx with Let's Encrypt.

7

It seems that everything on the domain ic0.app is not responding

8

Looks like the SSL certificates expired about 30 minutes ago?

9

Indeed seems so (I’m on GMT+2)

Capture d’écran 2021-10-13 à 15.11.04
Capture d’écran 2021-10-13 à 15.11.04984×564 73.6 KB

10

Alright it seems it has just been renewed. Services seems accessible too.

Capture d’écran 2021-10-13 à 15.13.27
Capture d’écran 2021-10-13 à 15.13.27984×654 72.4 KB

11

I’m excited for the post-mortem on why this happened

12

Ahoy ICP folks!

I can confirm that the issue has been fixed: InternetComputer Status - Boundary Nodes Connectivity Issues

and that we are working on an incident report (still investigating and confirming root causes).

13

Thx for the feedback @diegop :+1:

There seem to be quite some delay between the status and the effective issues, I mean, while the issue was effectively happening, the status page was still displaying zero problems and was displaying that everything was alright.

Is this also known and tracked?

14

Definitely. It was certainly unfortunate and not the experience we all want for the IC.

15

Thanks, looking forward to the details. In addition to understanding the technical issues, please unpack the implications from a single-point-of-failure “off-switch” perspective.

16

Incident report: