Hi all - a few people have asked about why this wasn’t caught during testing. This is a great question, and of course one that immediately came up on the team as well.
The basic answer is that there wasn’t an existing test in place and the same mistake that led to the error also resulted in not having a test.
The data migration got missed due to human error. Existing tests couldn’t have caught something about the new field that was introduced (because it wasn’t there). New tests ensured the new field was populated and relied upon in every case moving forward.
But because the data migration was missed, the need to test the migration was also missed.
So of course then we asked:
- How do we make it less likely we forget things in the future?
- Is there any useful test that could catch something without that something being well-defined?
- How could we make the eventually-inevitable errors that do happen less impactful?
We started addressing these internally and have other avenues we’re exploring, but so far we have agreed on the following responses to the above questions:
- To address the first point, we are creating checklists to make sure we remember to ask all the right questions when planning out the work as well as doing reviews.
- We thought of a test we could do against metrics on canisters with data to simply check for large deviations before and after upgrades in our state machine tests. This will likely catch large anomalies (such as many proposals getting deleted).
- We are working to prioritize canister snapshot support for NNS-controlled canisters along with our already-planned move of proposals into stable memory so we can stop garbage collecting them.
We do our best to avoid bugs, but in any system of sufficent complexity, they will happen. We will keep working to make them less frequent, and more importantly, less impactful when they happen.
We’re sorry this happened, but we’ll use it to make the IC even better moving forward.