A third faster, and fewer false alarms, in our integration tests
Every change to LedgerTM passes the full test suite before it merges. The slowest part is the component suite: about 4,600 tests across nearly a thousand test classes, each running real commands against a real database. On 30 September 2026 we finished a trial of a new way to run it.
The problem. Two costs, with one root.
- False alarms. Sometimes a merge failed on a test that had nothing to do with the change. We kept a list of known flaky tests and a rule: two unexplained failures and you stop debugging. A failure we cannot explain costs a merge, and often a session.
- Memory. The suite once grew to 26 GB of memory and the machine killed it. We capped it, but a cap hides the symptom. It does not say which test is growing.
Both come from one thing. Every test class ran in one process, sharing one message host. A leak in one test showed up as a failure in another.
What Bobcat is. Bobcat is an open-source test supervisor from JasperFx, the makers of Marten and Wolverine. It already runs Wolverine’s own test gate. It splits tests by class across several worker processes, each with its own environment. It retries a failure in a fresh process, and reports clean passes separately from passes on retry. It watches each worker’s memory and detects a stalled test.
We had hand-built half of that ourselves, piece by piece: a re-run script, a flaky-test list, an overnight flake hunt, a memory cap. Bobcat does it as one tool.
The trial. We ran five rounds of each shape on our test server:
| Runner | Component suite |
|---|---|
| Today’s runner, one process | baseline |
| Bobcat, 2 workers of 4 threads | 36% faster |
| Bobcat, 4 workers of 1 thread | 22% faster |
Neither Bobcat shape produced a new flaky failure. Every failure across all fifteen runs was one we already knew.
What we decided. We are adopting the two-worker shape, with a ceiling of 20 GB for the test processes. Adoption is its own piece of work, and first we split our largest test project so the workers balance.
Why it matters. A faster gate means more changes land each day. An honest gate means a red result is worth reading. Both come from a tool the framework’s authors maintain, rather than one we keep alive ourselves.
Written by Jeremy Rich.