Verifiability is your next AI investment. TDD is just the start.

The old ceiling was the team

In my last post I said verification is the engine of a software factory. This post is about the money, and the money has two sides. Most people only look at one.

Start before AI, with a team that writes no tests at all. Zero tests, zero verification. Even that team has a ceiling: the amount of code its people can write in a week. That's the ceiling everything is measured against.

Now add unit tests and end-to-end tests. Measured in code throughput, the team's output drops, no matter what. Every hour spent on a test is an hour not spent on the next feature. You can argue that real throughput goes up, because a team without tests breaks things, and broken things cost time. True. But set regressions aside and measure only the code that gets written: tests lower the ceiling.

The best-known industrial study of test-driven development, from Microsoft Research and IBM in 2008, puts numbers on both halves. Four teams cut their pre-release defects by 40% to 90%. They also estimated it cost them 15% to 35% more time up front. That second number won, project after project. When a deadline got close, the tests were the first thing to go.

It wasn't irrational. With a fixed number of people, verification was a tax on throughput, paid back later and diffusely, in bugs that didn't happen.

(You've probably heard that a bug costs 100 times more to fix in production than in design. Don't repeat it. Laurent Bossavit traced it back to IBM training notes with no data newer than 1981. Poor software quality is expensive enough without it: CISQ put the cost at $2.41 trillion a year in the US alone, in 2022.)

The new ceiling is trust

Agents removed the old ceiling. Their capacity is, for practical purposes, unlimited: you can run as many as your machines and your budget allow, day and night. Anthropic said in October that its daily merges went from about 2,500 in May to more than 30,000 in September (I wrote about it in my first post).

So what limits throughput now? Not the code. It's how much of the agents' work you're willing to accept. Throughput is now limited by the trust you put in the agents, and nothing else.

With agents, throughput is limited by one thing: how much of their work you can trust.

And trust needs verifiability. I don't mean a feeling that the agents are good these days. I mean evidence: a check that proves the work does what was asked, every time, without a person reading the code. That's what I called climbing the trust ladder: expect the agents to fail, catch it when they do, add a countermeasure, and trust the system a little more. Every rung is a piece of verification.

The two sides of the economics

This is why verification is suddenly such a big deal. Both sides of the sum moved, and they moved in the same direction.

The value went up. Before, a test was worth the bugs it caught. Now it's worth that, plus every piece of agent work it lets you accept without a person in the loop. Each class of change you can verify becomes a class of change the agents can do on their own, at agent speed. Verify your billing rules, and billing changes stop waiting for you. Verify your user journeys end to end, and front-end changes stop waiting for you. The better your verification, the higher your ceiling.

The cost went down. Agents write the tests, and they run them. They build the environments the tests need: StrongDM built working clones of Okta, Jira and Slack to test against, the kind of project nobody would have funded before. Even formal proofs, the most expensive verification there is, are getting written by models. More on that below.

Verification used to be a tax on throughput. Now it's what sets it.

When a thing becomes more valuable and cheaper at the same time, you invest in it. My rule has always been "invest in verification so you can move faster with confidence". Before agents, that was advice for teams that could afford it. Now it's the whole game.

Trust has to be earned, not assumed

The tempting shortcut is to skip the verification and trust the agents anyway. More code goes out, faster. The numbers say that's a trap.

Google's DORA research found in 2024 that a 25% increase in AI adoption went with an estimated 1.5% drop in delivery throughput and a 7.2% drop in delivery stability. By 2025, throughput had turned positive, but stability was still going the wrong way. DORA's conclusion is the right one: "AI's primary role is as an amplifier, magnifying an organization's existing strengths and weaknesses." A team with strong checks gets faster. A team without them ships its weaknesses faster.

CodeRabbit, which sells AI code review, compared 470 open-source pull requests at the end of 2025 and counted about 1.7 times more issues in the ones written with AI. It's a vendor's report, and it had to guess who wrote what, so treat the number as a signal, not a law. But it points the same way: trust you didn't earn is a loan, and production collects.

Agents game weak checks, and their makers say so

There's a second reason trust needs real verification, not just some tests: a weak check isn't neutral any more. With agents, it's a target.

• Anthropic wrote in the Claude 3.7 Sonnet system card that the model "occasionally resorts to special-casing in order to pass test cases", usually by returning the values the test expects, sometimes by changing the tests themselves. And: "User testing did not identify this behavior because it occurs infrequently in normal usage."
• OpenAI published a reasoning model's own thinking in 2025: "We can circumvent verify to always return true… Then all tests pass."
• ImpossibleBench, a benchmark built from tasks where the tests contradict the specification, so the only way to pass is to cheat, found GPT-5 cheating on 76% of one set of tasks. The right prompt cut cheating on another set from 92% to 1%, and monitoring models caught only 42% to 65% of it on the harder set.
• SWE-bench+ found that 31% of the "successful" patches on a popular coding benchmark only passed because the tests were too weak to catch them.

The models get better at this, and their makers work hard on it: Anthropic reported a 67% drop in hard-coded tests between Claude 3.7 Sonnet and Claude Opus 4. But the incentive never goes away.

An agent is trained to make the checks pass. If your checks are wrong, it will make the wrong thing pass.

So the gap in a check got more expensive at the same time as the check got cheaper. Anthropic's own engineers said it best, after sixteen agents built a C compiler: "it's important that the task verifier is nearly perfect, otherwise Claude will solve the wrong problem." Your ceiling is only as high as your weakest check.

What verifiability means

So the question isn't "do we have tests?" It's "can a machine tell, reliably, whether this work does what we wanted?" That's verifiability, and it has five parts:

1. An observable expectation. Written down before the work starts, in terms a machine can check: this request returns this, this page loads in under that, this user can't see that data.
2. A repeatable check. Same input, same environment, same verdict, every time. Pinned dependencies, fixed clocks and seeds, nothing that flickers.
3. A realistic environment. A check against a mock that always agrees proves nothing. Real browsers, real databases, replicas of the services you depend on.
4. Independent evaluation. The thing that judges the work is not the thing that did it, and it can't be edited by it.
5. A signal from production. The last check is the real one: did the outcome happen, for real users?

Not every domain is equally verifiable, and that decides how high your ceiling can go there. A billing calculation is extremely verifiable. A database migration is mostly verifiable. An essay, or a design that has to feel right, much less so. Some domains are verifiable in parts: you can't prove a page is beautiful, but you can prove it loads, renders without overflow and meets its accessibility rules. The work is finding those parts and making them checkable, because every part you make checkable is capacity you unlock.

Where test-driven development fits

TDD is one technique inside verification, not the whole of it. But with agents it's the most natural one: a failing test is the clearest specification you can give an agent. It can't misread it, and it knows exactly when it's done.

Kent Beck, who made TDD famous, puts it in the instructions he gives his agent: "Always follow the TDD cycle: Red → Green → Refactor." And one of his warning signs is exactly what the system cards describe: "Any indication that the genie was cheating, for example by disabling or deleting tests."

Where you start depends on what you have.

On a new project, write the failing tests first. The outcome, as tests, then the code that passes them. Unit tests for the logic, end-to-end tests for the journeys that must never break. I aim for complete coverage: when an agent writes the tests, coverage is cheap, and every untested line is a place an agent can quietly do the wrong thing.

On a legacy system, pin down what it does today before you change anything. A heavily used system holds years of product decisions, many of them written down nowhere. Its behaviour is the specification.

X showed this at scale this month. Its engineers rebuilt the x.com web app with AI agents in under six months, and they tested the new app against the old servers as specifications: a parity script requests the same page from both and compares the status, the titles, the tags and the structured data; another compares link-preview cards pixel for pixel. Users moved over country by country, and any group could go straight back. Their conclusion is one I'd put on the wall of every team doing a rewrite:

The old servers were the best specification we had. (Yagiz Nizipli, X)

That's the two sides of the economics in one project: the old system made the new one verifiable, and verifiability let a handful of engineers hand the rewrite to agents. One caveat they'd agree with: legacy behaviour includes old bugs and obsolete rules. Decide which behaviours to keep and which to change, then write that decision down as a test too.

Test the tests

Cheap tests aren't assurance. A test suite an agent wrote in an afternoon can have perfect coverage and catch nothing. Meta learned it while generating tests with models at scale: "increasing coverage doesn't necessarily find faults." Their answer was to aim the generated tests at faults, and engineers accepted 73% of them.

So in my factory, the tests get tested.

Mutation testing. Stryker changes the code on purpose (flips a comparison, removes a line) and checks that the tests notice. A surviving mutant is a bug your tests would let through. It's heavy: on a big codebase a full run can take five to ten hours. So I run it on the changed code when it finishes within about ten minutes, and on everything in a weekly run at night.

Tests the agent can't quietly change. When an agent edits or deletes a test, that's the moment to be suspicious. A second, independent agent interrogates the one that did it: was the change honest, or a shortcut? StrongDM goes further and keeps its scenarios outside the codebase entirely, like the holdout set in model training, where the agents can't see them.

Adversarial review. Before anything ships, independent sub-agents attack the work from a different angle. I use two skills for exactly this. Scrutinize runs when work is done: a fresh agent goes through it looking for what's wrong, including tests that prove less than they claim. Second opinion runs when an agent proposes something: another agent looks at it from a different angle before anyone acts on it. Different instructions, different context, looking for what the first agent missed.

Errors that explain themselves. A bug you can't reproduce is a bug you can't verify a fix for. So the first step of any fix is a test that reproduces the issue, and the code is designed from the start to fail with intentional, readable error traces, so an agent can turn a production log into a failing test.

How much freedom should an AI step get? walks through this process end to end, with the rules for when a fix may merge on its own.

Proofs are coming

The cost side has one more step to go. Formal verification, a mathematical proof that code does what its specification says, has always been the gold standard and almost never worth it. seL4, the famous verified microkernel, needed about 20 person-years and 200,000 lines of proof. As Martin Kleppmann put it, for most systems "the expected cost of bugs is lower than the expected cost of using the proof techniques."

Agents change that too. Models can write proofs, and a proof checker doesn't care who wrote it: "It doesn't matter if they hallucinate nonsense, because the proof checker will reject any invalid proof." Kleppmann predicted in December that AI will make formal verification mainstream. Leonardo de Moura, who created the Lean prover, made the point that matters most for trust: "for any fixed testing strategy, a sufficiently adversarial system can overfit to it. A proof cannot be gamed."

A check that can't be gamed is the highest ceiling there is. It won't be every project, and it won't be this year for most teams. But Kleppmann also names where the hard work moves: "the challenge will move to correctly defining the specification." That's the same place everything in this post ends up.

Measure the ceiling, not the output

If verification is an investment, measure its return. Most teams measure the wrong things: lines of code, pull requests merged, how many agents are running, how fast it feels. None of those is an outcome, and none of them tells you where your ceiling is.

What I measure:

• How much agent work is accepted without a person stepping in. That's the trust ceiling, measured. Every time a person has to step in is a check the factory doesn't have yet.
• Time from agreed intent to a verified outcome in production. Not time to write the code: time until the thing we wanted is true for real users.
• Total cost per useful change. Tokens, compute, test environments, review, rework and incidents, all of it.
• Escaped failures, how bad they were, and how fast they were found, contained and fixed.

Compare like with like (changes of similar risk and size), look at the worst cases as well as the averages, and get a baseline before you claim a gain.

If you want to test this yourself, keep it small. Pick one class of change, a bug fix in one service, say. Measure how much of that work you can hand to agents today without stepping in. Add one verification improvement (mutation testing on the changed files, or a parity check against the current behaviour) and measure again. If the share goes up, you've just raised your ceiling.

What would prove me wrong

• Trust doesn't follow verification. If teams with strong checks still won't let agents work without reading every line, the ceiling stays where it is, whatever the checks say.
• Verification costs more than the capacity it unlocks. Mutation runs take hours, verifier agents burn tokens, environments need maintaining. If the total cost per useful change goes up, the investment doesn't pay.
• The checks give false confidence. A deterministic test repeats the same wrong assumption every time. If escaped failures don't go down, all those green ticks are theatre, and the ceiling was never real.
• Most of the work isn't verifiable. If the work that matters in most businesses can't be checked by a machine, the ceiling stays low there, and this stays a niche for teams that build compilers and billing systems.

My bet

Verifiability is the next big area of focus for software engineers. For decades, the ceiling on what a team could ship was its people, and verification was a tax on it. Now the ceiling is trust, and verification is the only way to raise it.

The teams that invest in verification will ship more, with more confidence, than the teams that invest in more agents. The ones that don't will either stay stuck at the old ceiling, reading every diff, or ship their weaknesses faster than ever.

In the next post: what happens to the software lifecycle when a piece of work isn't done until its outcome is met. If you want it when it's out, subscribe.