If your test agent reads the code before it reads the spec, it is grading the implementation against itself. I've suspected this for a while, every time an agent-written test suite came back green on a feature I knew was subtly wrong. Spec-Driven PBT (Shukla, Gupta, Alavi and Vaezipoor, posted 11 November) is the first paper I've seen that measures it on whole agent-built applications, and the result is more interesting than "specs good, code bad".
The setup is a straight contrast. Both arms run Claude Opus 5 inside the Claude Code harness, writing Hypothesis property-based tests against 25 ViBench applications, each built by a coding agent from a product requirements document. The baseline is an adaptation of Anthropic's Agentic PBT method: read the code, infer invariants, write tests, triage failures. The new agent does the same loop with one hard rule. It writes its invariants into an invariants.md file from the PRD first, and is told not to open the codebase until that ledger exists. Only then does it read the source, to find inputs and operation sequences that might break those invariants.
Tests written from the code inherit the code's opinions#
The core argument is the classic test-oracle problem, restated for agentic coding. The authors put it in one line:
a rule that the code implements consistently but incorrectly will pass every test derived from it.
That matters more now than it used to. For a mature library, docstrings and call sites encode years of intent, so code is a decent oracle. For an app an agent generated last Tuesday from a PRD, the code's only claim to correctness is the agent that wrote it. The authors also note they tried giving the agent the spec and the code together, and spot checks showed it still wrote tests tightly coupled to the implementation. Access to the spec wasn't enough. Ordering was the fix.
Same budget, half the overlap#
Here are the ViBench numbers from the main table:
Spec-Driven wrote 566 property tests; the code-driven baseline wrote 1,384.
34% of spec-driven tests were tied to at least one candidate bug, versus 13% for the baseline.
Candidate bugs per test: 0.22 vs 0.08, and spec-driven had the higher yield on all 25 applications.
Total candidate bugs: 126 vs 113. Of 169 distinct bugs, 70 were found by both, 56 only by spec-driven, 43 only by the baseline.
Cost per PRD was nearly identical: $8.22 vs $7.69 mean, about 31 vs 30 minutes of agent time.
The headline isn't that one method wins. Neither does, really. The overlap is only 41% (Jaccard), and nearly 60% of the bugs were caught by just one of the two agents. For roughly eight dollars per app, you get two mostly separate bug lists.
The kinds of bugs differ too. State, lifecycle and logic failures made up 61% of the bugs only the spec-driven agent found, against 31% of those only the baseline found. Input handling went the other way: 49% of the baseline's exclusive bugs, 16% of the spec-driven agent's. Across three repeated runs per method the split held, so it isn't one lucky seed.
The examples make the split concrete. The spec-driven agent took the PRD line "following a public account takes effect immediately" and drove two accounts through six prior relationship states. One failed: a follow request left pending from when the target was private stays pending forever after the account goes public. In a GitHub-like app, it turned "history is append-only" into a race test and found accepted commits silently disappearing with just two concurrent writers. The commit handler had no docstring and no lock, so the code-driven agent had nothing there to check against. Meanwhile the code-driven agent found what you'd expect from reading routes: a bare int() on a cart quantity that returns a 500, and a username that gets Unicode-normalised after validation, so the user can't log in. No PRD would ever mention that.
The kick button that shows who the oracle is#
The case I keep coming back to is a bug both methods found. In a Discord-like app the kick-permission rule is implemented twice: once in the flag that shows the kick button, once in the endpoint that performs the kick. They disagree when the two members have equal rank. The code-driven test checked that the two copies agree. The spec-driven test first checked the endpoint against the PRD, which says equal rank does not outrank.
Both tests failed. But the authors report that the code-driven agent's suggested fix would have changed the endpoint to match the flag. That would make the test pass while breaking the requirement. The spec-driven fix changed the flag.
Without an external oracle, whichever copy the agent read first became the truth.
This is the part that should worry anyone running a fix loop of the form "agent writes tests, agent makes tests pass". It's one anecdote, and the paper calls it that. But it's precisely the failure mode that never shows up as red in CI.
Spec-only lost, which is the most useful result in the paper#
Appendix F is where I'd point skeptics, including myself. In a controlled comparison where invariants were frozen from either the PRD alone or the code alone, the spec-only arm did clearly worse: 78 candidate bugs against 117, with a yield of 0.11 against 0.23. The overlap dropped to 23%. So "just derive tests from the spec" is not the lesson. Spec-first, then code to refine and find falsifying inputs, is what produced the main result. The authors are also upfront that the two main arms differ in prompt and workflow, not only in grounding, so they can't say how much of the efficiency gap the spec alone explains.
There are other reasons to hold this loosely. ViBench has no ground-truth bug labels, so every "bug" is a candidate judged by a blind Claude Opus 5 reviewer. Two authors checked 25 of them by hand and all 25 were real, but that's a sample, not the corpus. On PBTBench, where the "spec" is API reference docs, the overlap between methods jumps to 84% and the advantage nearly disappears (88% vs 85% recall). Their own explanation is that reference docs describe function contracts rather than product behaviour. And the ViBench PRDs are thorough. The authors expect the method to degrade with a thin or stale spec, and I'd expect that too.
Two test agents, one ledger#
Here's what I'm changing. When I build a feature from a spec in Claude Code, the verification step is now two sub-agents, not one. The first gets the spec and a hard instruction to write invariants.md before it touches the repo. The second is a plain code-reading property tester. They run in parallel, write to the same output directory, and I review the union. At roughly the same cost per run, the paper says you get two different slices of the bug space. That's a cheap trade.
The ledger is also the review artifact I actually want. A list of natural-language invariants is short enough to check against the spec in minutes, without reading 1,400 tests. If an invariant is wrong, I fix the spec or the ledger, not the code.
My bet: within a year, "which artifact is the oracle" will be an explicit setting in agent harnesses, the way permission modes are today. Any loop where the same agent reads the code, writes the tests and proposes the fix should have to say where its idea of "correct" came from.