Boundary Nodes Connectivity Incident Retrospective - Wednesday October 13, 2021

On 2021/10/13 because the systems which were to deploy a renewed certificate for ic0.app had failed, the certificate expired resulting in the inability of web clients to authenticate the certificate and in ic0.app becoming inaccessible. This was not reflected on the status page. The problem was resolved by deploying the renewed certificate. See the attached graph which shows the event and recovery.

This event did not affect the Internet Computer blockchain proper but rather access to it via the TLS secured ic0.app domain. To ensure that this does not happen again and to address the underlying issue of access via other paths and domains we have a number of short and longer-term mitigations and proposals.

In the short term, we have corrected the systems which deploy the ic0.app certificate, we are adding additional probes and monitoring to ensure quicker status updates and response and we will follow up with a post detailing how to access the Internet Computer without depending on the ic0.app certificate to eliminate a “single point of failure”.

In the longer term, we have plans for architecture changes to increase the resilience and independence of Internet Computer blockchain access including increased number and diversity of boundary nodes and alternate discovery, authentication, and confidentiality mechanisms as we free ourselves from the limitations of conventional web technologies.

Please watch the forum, make and vote on NNS proposals and help expand and improve the Internet Computer blockchain.

10 Likes

Thanks for the retrospective! Is the boundary node code open source yet?

2 Likes

Parts are already open source (e.g. the service worker and the icx-proxy). Other parts are coming out soon (the control plane). The rest will come out as soon as we can get it cleaned up.

2 Likes

Great, thanks for the clear explanation and roadmap.

2 Likes