AI writes tests faster than we can run them
Everybody is talking about how much faster AI makes a developer, and it does. What gets mentioned less is where the time goes afterwards. An AI session will happily write forty tests for a change where I would have written six, and they’re good tests, and every one of them has to run before the next change can merge.
Between 3 and 18 October our integration tests went from 3,394 to 6,680. They doubled in fifteen days, while the codebase itself grew 18%. The stage that runs them went from six and a half minutes to nearly nineteen.
Half an hour a change
Every change passes a gate before it merges, and the gate ran on the Linux box at home, one change at a time. By the middle of October a gate took about thirty minutes. We were merging around fifteen changes a day, which meant the box spent more than seven hours a day doing nothing but gating, and changes queued behind each other. One change had to wait twenty-five minutes before its gate even started. Another, through a combination of our own mistakes, was proved three times over and took two hours and twenty-six minutes to land.
We run several lanes of work at once, so a slow gate doesn’t cost one person half an hour. It backs up everything.
More workers made it slower
The obvious fix was to run more tests in parallel on the box, so we tried it. Three workers took 1,028 seconds. Four took 1,127, and five took 1,319. The box has 32 GB of memory and every worker needs its own database and its own copy of the application, so past three they were fighting each other for memory. That was a lesson. We’d been assuming we were short of processor, and we weren’t.
Rent the machines, build once
So we rented. Ubicloud hires out build runners by the minute, and their 16-core machines ran our integration suite in 690 seconds against 842 to 926 on the box. We split the tests into three slices, each on its own runner, using the measured time of every test to balance them, with the longest placed first.
The first version had each runner check out the code and build it. When we looked at the bill, each slice was spending about four and a half of its eleven minutes compiling, the same code, three times over. That was 40% of what we were paying for.
Our first attempt to fix that was to build once on the box and send the result to the runners as a file. The download alone took between 193 and 350 seconds, which was slower than building it again, so we threw that away. The second attempt was to package the build as a container image and have the runners pull it, and that takes about forty seconds all in. A slice’s test stage dropped from over ten minutes to between four and seven and a half.
So the process now is that the box builds once, publishes the image, and three rented runners pull it and test in parallel while the box gets on with the cheaper checks. It costs $0.016 a runner-minute. October came to R294 for 33 gates, and we’ve capped it at R2,200 a month. If the rented side misbehaves, one setting sends everything back to the box.
Where we actually are
A green gate now takes between seventeen and nineteen and a half minutes. That’s a real improvement on thirty, and it’s well short of the ten to twelve we were aiming for.
In November we tried six slices to see if more parallel runners would close the gap. It cost about R2 more per gate and saved a minute and a half, so we went back to three. The slow part has moved. It’s now the build and the checks on the box, at seven to nine minutes and then another six to eight, and no number of rented runners makes those quicker.
There’s more that isn’t right. The box is still in the path of every gate, so if it’s down, nothing merges. When we hit the day’s spending allowance the tests fall back to the box at roughly twice the time, which tends to happen on exactly the busy afternoon when we least want it. And we spent a fair amount of effort on running only the tests a change could affect, which turned out to be safe and nearly useless: it narrowed the run 12% of the time against a target of 30%, so we didn’t build the rest of it.
The suite, meanwhile, has kept growing. It passed 13,600 tests on 9 November. We’ve started moving rule checks into fast unit tests, which is probably the real answer, because the fastest integration test is the one you didn’t need.
Written by Jeremy Rich.