What the Linux box does while I’m asleep
The gate that every change passes before it merges has to be quick, so it can’t run everything. The slow tests, the coverage measurement and the long simulations were things we ran when somebody remembered to, which in practice meant after something had already gone wrong. Since October they run every night, on the Linux box at home, starting at one in the morning South African time.
We did try letting GitHub start it on a schedule. GitHub ran the midnight job about six hours late, two nights in a row, so now the box has its own timer.
What runs
First, every unit and integration test we have, including the ones marked as too slow for the gate, with coverage measured as they go. Coverage here means more than a percentage. The run works out which handlers no test reaches at all, and which are reached by only one, and writes a ranked worklist. One of the early nights found eight API endpoints that no test touched.
Then a security scan, a check for out-of-date packages, and some housekeeping on the box itself, which is less glamorous but matters, because the box got to 95% full one night and that was the end of that run.
Then the part we call Prove. It takes the sample book for every template and restores it twice, once on the last released version and once on today’s code. Both sides then walk the same window, from the middle of November to the middle of February, which is about ninety-two day-ends and takes in three month-ends, a quarter-end and the year-end. At the end it compares the two books. If today’s code produces a different balance, a different fee or a different journal from the release, we know about it in the morning, and either we meant it or we didn’t.
Each step runs even if the one before it failed, so one bad stage costs its own section of the report and not the whole night.
The night shift
When we first built this, the rule was that nothing overnight edits code. The run reported, and a person fixed. After a couple of weeks of reading the same kinds of small failure every morning we changed our minds.
The night shift now takes red tests on the main branch, flaky tests, findings we have tagged as small, and coverage that has slipped. It works under rules that are tighter than the ones we work under ourselves. It takes at most ten items a night. Every fix is its own commit, naming the item and the test that proves it, and the lot goes through the same gate as any other change. It gets two attempts at an item and then puts it on a “could not fix” list. It may never change what a test says the product should do, and a check at the end reverts anything that deleted a test, skipped one, or changed an expected value, because the easiest way to make a red test green is to stop it testing anything.
Product decisions, design and deployment stay with a person.
The morning
At seven a script builds a digest of the night, and a short AI session reads it and writes the morning report. Findings that a landed commit names as fixed are closed without anybody asking, and old ones are parked.
Running everything twice over, round after round, is also how we hunt flaky tests. One long hunt did 152 full runs of the suite, 13 went red, and 12 of those 13 traced back to a single shared variable in one test helper. That was a good night’s work for a machine.
Where it’s still weak
For a week in the middle of November the night shift fixed nothing. The service that starts it had been set to start in the wrong folder, so the tool it uses to look up the night’s results couldn’t find the repository, and every night it waited four hours for a report that was already there and then went home. It took us six nights to notice. It’s fixed, and it now stops at once if the lookup fails.
The digest also tells us what failed without telling us why, which means the morning still starts with some detective work. And Prove gave no verdict at all on two nights in October. We’re working through those.
Written by Jeremy Rich.