Journal · How we build · 17 November 2026 · 22:48 SAST · by Jeremy Rich

Measuring the factory, not just the product

The usual way to find out a build is slow is that somebody complains. You spend an afternoon with a stopwatch, fix whatever is most obviously wrong, and forget about it until the next complaint.

That doesn’t work when most of the building is done by AI sessions, because they don’t complain. A session will wait three minutes for a build forty times a day and never mention it. So we started measuring the development process the way we’d measure a production system, and it’s been one of the more useful things we’ve done this year.

What we collect

It began in September with a script that reads what’s already lying around: the git history, the findings queue, the log of work handed to each lane, the test timings from the gate, and the session transcripts. A small hook also logs every question a session asks me, because the number of times I’m interrupted turns out to be a very good measure of how well a piece of work was specified.

Since then we’ve added to it. Every gate run keeps its stage timings. Every test has its own measured time. Every lane writes a row when it stops, with the model it used, how its time split between thinking, building and waiting, and what it cost in tokens. Every landed change records its gate minutes and how many times it went red on the way.

All of it lives in plain files on the machine that did the work. We decided against a telemetry service or a database, on the grounds that we’d then have one more system to keep running in order to find out why the other ones were slow.

Once a day a retrospective reads the numbers, names the one that moved the wrong way, and proposes one change. It was weekly at first, and a week turned out to be a long time to keep doing something wrong.

What it found

The cheapest win was in the build itself. Lane builds had a median of 3.3 minutes, which seemed high for a small change. The timings showed that every commit rebuilt all 121 projects, because two well-meaning features were stamping the commit id into every build. We turned that off for development builds, and an empty commit followed by a build went from 168 seconds to 7. Not bad for a two-line fix that nobody would ever have gone looking for.

It showed that the test suite had doubled in fifteen days while the code grew 18%, which is how we knew the gate problem was a test-count problem and not a slow-machine problem. It showed that 40% of what we were paying for rented runners was spent recompiling code we’d already compiled.

And it showed what a red gate was really worth. In one week in November sixty gate runs failed. Seven of those had found a real defect. Thirty-one were bookkeeping, things like a stored snapshot that needed regenerating, and fourteen were noise. So we changed the gate to retry a failed test once on its own in a fresh process, and to fix the bookkeeping before it pushes rather than fail on it afterwards.

When the instrument was wrong

The telemetry was also wrong, and the humbling part is how long it took us to notice. One lane recorded 36 minutes of model time over a 32-hour piece of work. When we went back to the transcripts they held 523 minutes. The lane had done its building in sub-sessions, and we’d been counting their tokens and ignoring their clocks, so a lane that was flat out read as idle. For about a week another counter read zero for every lane and nobody questioned it.

The numbers also disagreed with us a couple of times. We were sure more parallel test workers would be faster, and the timings said four were slower than three and five slower still. That was memory, not processor.

What’s still missing

Until this week the records were scattered. Builds on my Mac, builds on the Linux box and builds on the rented runners each kept their own, and the ones from the rented runners were simply lost when the machine was handed back. As of 17 November every build writes the same labelled row wherever it runs, and the rows come home to one store.

That’s the collecting sorted. The reporting has not caught up: the dashboard still shows only part of it, and the daily numbers on the Mac still read only the Mac’s work. So we now have a good deal more data than we have answers, which I suspect is the normal condition.

Written by Jeremy Rich.