The test is written before the code, and by somebody else
Anybody who has run a software project knows the conversation where the developer says it’s finished and the business says that isn’t what they asked for. Both of them are usually right. The requirement was a paragraph, the developer read it one way, and then the same developer wrote the tests, which naturally agree with the developer.
AI doesn’t make this better. It makes it faster. An AI session that builds a feature and then writes the tests for it will produce a lovely green result, because it’s marking its own homework. Our acceptance criteria used to be ten to thirty-five words of prose per piece of work, and the session that built the piece was also the one that judged it.
Scenarios in the design
So we put a specification step into planning. Any design that changes how the product behaves now has to carry its scenarios, in the Given, When, Then form that a banker can read and argue with:
Scenario: A renewed mandate is valid again
Given a mandate that lapsed on 2026-11-30
When a banker renews it until 2027-11-30 and a second banker approves
Then the mandate's standing is InForce
They’re written while the design is still a proposal, so they get reviewed with it. The design is where it’s cheap to find out that we’ve misunderstood something. There are now more than four thousand of these scenarios across a couple of hundred designs.
We keep them inside the design document and not in separate feature files. We tried to imagine maintaining both and decided we’d be lying to ourselves.
A different author
When a design is planned, each scenario is mapped to a named test, and a separate AI agent writes those tests. It runs before any of the code exists, and it never sees the implementation, only the scenarios and the questions it’s allowed to ask. The rules it follows are simple. A Given becomes a piece of set-up, a When becomes exactly one action, and a Then becomes a check.
The session that builds the feature is handed tests it didn’t write and can’t quietly bend. If it thinks a test is wrong it has to say so and get a ruling.
I should be straight about one thing. We decided on purpose not to build a mechanical generator that turns scenario text into test code. An AI author writes the tests, and what keeps it honest is a check, which is the next part.
Bobcat holds the two together
We moved our integration tests onto Bobcat at the start of October, and later that month we adopted its way of writing tests as well. Each test carries the scenario’s title, and each step in the test carries the text of the Given, When or Then it implements. When Bobcat runs, it writes out the steps it actually executed.
A script then compares those with the scenarios in the design. A step that appears on one side and not the other stops the change from landing. So the specification can’t drift from the test in either direction.
A new test arrives marked as an acceptance test and is promoted once it has proved itself, and a failing one now reports every check that failed and not just the first.
What we don’t know yet
We believed a separate author would write better tests than the builder would, and we shipped it on that belief. We still don’t have the numbers to prove it, and I’d rather say so than pretend.
The other thing is that it works rather too well. Scenarios are easy to write, so we write a lot of them, and each became an integration test against a real database at a bit over a second apiece. By the middle of November there were 12,905 integration tests and that stage of the gate took about twenty-nine minutes. We’re now moving the proof of pure business rules down into fast unit tests and keeping one end-to-end test per scenario. The first batch retired seven integration tests where we’d predicted about a hundred, so that is not finished either. More on the speed problem here.
Written by Jeremy Rich.