A generated test suite that passes on the reference implementation 96% of the time sounds like a good test suite. In TestSuiteBench, Claude Opus 5 hit exactly that validity figure, and the same suites still killed only 55.5% of the injected bugs. The weakest model in the study killed 19.3%.
The paper comes from Egashira, Mündler-Sasahara, Vechev and Müller. The setup is clean: 165 open-source command-line tools in Rust, Go and C/C++, each with a natural-language spec, a reference implementation and about 74 mutants on average. The agent gets only the spec and has to write a complete test suite. Six models ran with vendor harnesses and a 6-hour timeout.
Valid is not the same as strong#
Three metrics matter. Validity is the share of generated tests that pass on the reference. Mutation score is the share of mutants the suite catches. The third checks the reverse direction, how many of the reference tests a generated implementation passes. Here is how the models landed:
Claude Opus 5: 55.5% mutation score, 96.1% validity, $10.56 and 37 minutes per instance
GPT 6 Astra: 50.1%, 94.3%, $2.18, 12.4 minutes
Gemini 3.8 Flash: 37.9%, 88.2%, $1.63
GPT 5.6 Sol: 37.7%, 94.4%, $0.89
DeepSeek V4 Flash: 28.6%, 78.1%, $0.14
GPT 5.6 Luna: 19.3%, 74.1%, $0.03
Look at Sol: 94.4% valid, 37.7% mutation score. Almost everything it wrote was correct, and it still missed most of the bugs. Validity only tells you the tests do not lie. It says nothing about what they leave out. The authors put it bluntly: agents can build software they cannot verify.
The failure mode is boring and familiar. Suites skip input combinations and edge cases, like testing a command with one file when it accepts several. Sol's suites were 79.9% smaller than the reference suites. That is not a model writing wrong tests, it is a model stopping early.
The usual fixes barely move the needle#
This is the part I would pin to the wall. Giving Sol the executable raised validity from 94.4% to 98.4%, which is nice, but mutation score went from 37.7% to 38.1%. Being able to run the thing makes the tests more correct, not more thorough.
Reasoning effort helped more: Sol went from 31.3% at $0.45 per instance (low) to 46.5% at $2.4 (xhigh). That is real, but you pay roughly five times as much for fifteen points. Model choice moved the number further than either lever, and model choice is also where the cost spread is wild: Opus 5 cost about 12 times what Sol did for 17.8 more points.
Spec length did not rescue anyone either. Correlation with mutation score was negative (ρ = −0.16 for Opus, −0.42 for Sol). The authors conclude the bottleneck is test design, not spec clarity. I find that convincing because the specs here are already unusually detailed.
Pooling three agents beats one, but not by much#
Of 12,130 mutants, 3,080 were killed by all of Opus, Sol and Gemini. Merging the three suites reaches 59% against 54% for Opus alone. So the misses are largely shared. Ensembling different models does not fix a blind spot they have in common, which matches what I see when two agents review each other's work and both nod.
Cheap models make the gap obvious#
The bottom of the table is a useful warning for anyone routing work to small models to save money. GPT 5.6 Luna costs three cents per instance and finishes in 4 minutes, and its suites catch under a fifth of the mutants while a quarter of its tests fail on the reference. DeepSeek V4 Flash at $0.14 is better but still only 28.6%. Cheap test writers are not a discount version of good ones. They produce a different thing, a suite that looks like a test suite.
That matters for how you budget. The expensive model's $10.56 is small next to an engineer's hour, and the 55.5% it buys is the only number in the table I would not be embarrassed to defend in a review.
What I would change in a spec-driven loop#
If your pipeline goes spec, then implementation, then agent-written tests as the gate, this paper says the gate is leakier than the green checkmark suggests. Three things I would do:
Stop reporting pass rate on agent-written tests as a quality signal. Track a mutation score, even a rough one, on the modules that matter.
Make the test-design step explicit. Ask the agent to enumerate input combinations and edge cases from the spec before it writes a single test, and review that list.
Spend the budget on the test agent, not the implementation agent, when the implementation is cheap to regenerate.
The reference-pass-rate numbers point the same way from the other side: generated implementations passed 65–93% of the reference tests, so the spec alone also leaves real behavior underdetermined.
Where the evidence stops#
Mutants are a proxy. Each one breaks a single requirement by design, so the distribution may not look like real faults, and the authors audited for hard-coded behavior but not for niche mutants. The specs are far more detailed than what most developers write, and a 30-instance check against developer docs dropped Sol's mutation score from 25.5% to 4.0%. So the numbers here probably flatter agents. Everything is also limited to compiled CLI tools, with the reference executable and the reference implementation's quirks treated as ground truth. The abstract and Section 3.1 also disagree slightly on the reference pass rate (82% versus 86–93% for most models).
My bet: within a year, mutation score or something like it becomes a standard column next to pass rate in every coding agent eval. Until then, treat a green agent-written suite as evidence the code is not obviously broken, and nothing stronger.