Boundary Nodes Connectivity Incident Retrospective - Wednesday October 13, 2021

Developers
1

On 2021/10/13 because the systems which were to deploy a renewed certificate for ic0.app had failed, the certificate expired resulting in the inability of web clients to authenticate the certificate and in ic0.app becoming inaccessible. This was not reflected on the status page. The problem was resolved by deploying the renewed certificate. See the attached graph which shows the event and recovery.

2048×821 151 KB

This event did not affect the Internet Computer blockchain proper but rather access to it via the TLS secured ic0.app domain. To ensure that this does not happen again and to address the underlying issue of access via other paths and domains we have a number of short and longer-term mitigations and proposals.

In the short term, we have corrected the systems which deploy the ic0.app certificate, we are adding additional probes and monitoring to ensure quicker status updates and response and we will follow up with a post detailing how to access the Internet Computer without depending on the ic0.app certificate to eliminate a “single point of failure”.

In the longer term, we have plans for architecture changes to increase the resilience and independence of Internet Computer blockchain access including increased number and diversity of boundary nodes and alternate discovery, authentication, and confidentiality mechanisms as we free ourselves from the limitations of conventional web technologies.

Please watch the forum, make and vote on NNS proposals and help expand and improve the Internet Computer blockchain.

12 Likes
2

Thanks for the retrospective! Is the boundary node code open source yet?

2 Likes
3

Parts are already open source (e.g. the service worker and the icx-proxy). Other parts are coming out soon (the control plane). The rest will come out as soon as we can get it cleaned up.

2 Likes
4

Great, thanks for the clear explanation and roadmap.

2 Likes
5

One of my devices still seems to fail to update the certificate, which gives me http error 500. Is there a way to manually fix this issue?

1 Like
6

Thanks for letting us know.

  1. Is this still happening?
  2. Can you share more about the device type? Are you seeing the certificate fail via browser or something else?
1 Like
7

Thank you for your reply.

  1. Is this still happening?
    Yes.
  2. Can you share more about the device type? Are you seeing the certificate fail via browser or something else?

Sure. I am using windows 10 desktop. I am using chrome but I have the same error with other browser like edge.
I have other devices: window 10 laptop and android phone. Both work fine.
Please take a look at screen shot of the browser error.

ss
ss1920×1080 112 KB

3 Likes
8

Thank you, let me ping the team on this.

1 Like
9

@birdynyamnyam One possibility is a stale certificate being returned from the chrome cache.

Please try if incognito mode loads the URL.
If the URL load in incognito mode, then clear the chrome cache.

3 Likes
10

I have already tried incognito mode. The page didn’t load. Then I have also cleared cache. Still page won’t load. I have tried a few non-chromium browsers but I get the same result. I am starting to feel like I have to reinstall whole machine lol.

2021年10月28日(木) 7:46 faraz via Internet Computer Developer Forum <dfn@discoursemail.com>:

1 Like
11

Thanks for letting me know. Ok, let me ping team again.

3 Likes