Hi everybody!
The bitcoin canister has been running successfully for more than half a year now. During this time, DFINITY has of course kept a close eye on whether everything was working as expected. We set up some tooling that queries the height of the Bitcoin canister, and compares it to what various block explorers have, such that alarm bells would go off if they disagree too much (which was never the case!). This monitoring currently relies on DFINITY’s involvement, and we thought we could improve things further by instead having this monitoring system run as an autonomous “watchdog canister” on the Internet Computer. We now made progress on this idea and would like to propose this more concretely to the community.
The watchdog canister (source on github) regularly queries a list of block explorers via HTTPS outcalls to obtain their view of the latest Bitcoin block height. More precisely:
- It queries all block explorers on the list to obtain their view of the latest block height.
- Some explorers may temporarily be offline, so it filters out all errors, and only continues with the successful results. If less than three blockchain explorers respond successfully, then the watchdog failed to establish the block height this time, and does not take further action until the next check.
- It defines its “target block height” as the median of all heights returned by the block explorers. It uses the median such that one block explorer returning an incorrect height cannot influence the target height.
- It compares its target height with the block height that the bitcoin canister currently has. If this differs by more than 2, something must be wrong. This would be a dangerous scenario as canisters that build on the Bitcoin integration (such as ckBTC) may now get inconsistent information from the Bitcoin canister, which can have serious consequences. We propose that the watchdog canister can pull the emergency brake in such a scenario, and temporarily pause the Bitcoin canister from responding to requests: it is better to return an error than to give information that may be incorrect. It would re-enable access whenever the Bitcoin canister and Bitcoin explorers agree again. If the divergence can only be fixed through an upgrade to the Bitcoin canister, then access to the Bitcoin canister would only be restored as soon as the upgrade proposal is accepted and the Bitcoin canister caught up to the latest Bitcoin height.
We view this watchdog canister as an additional safety mechanism, much like airbags in a car: we never expect this mechanism to kick in, and based on the historical data it would have never intervened in the half year since the launch of the Bitcoin canister, but in the unlikely event the Bitcoin canister and block explorers have a different view of the current height, it seems prudent to temporarily pause the Bitcoin canister access.
Unless there are concerns, DFINITY plans to submit proposals in the coming weeks to
- Install the watchdog canister under NNS control
- Update the Bitcoin canisters (mainnet and testnet) to allow the watchdog canister to pause and unpause its operations
Please let us know what you think! We’d be happy to hear your feedback and to answer any clarifying questions here.