Most of this morning's arxiv batch was scaffolds and benchmarks. One paper asked a much smaller question and got a far more uncomfortable answer: when your repair agent runs a test and it goes green, does that test actually touch the bug? Nobody had measured it properly. The answer is that roughly half the time it doesn't — and the agent closes the task anyway. Today's thesis: a passing test is not evidence until you know it fails on the broken code.
What it does#
The paper is Validation Evidence in LLM Repair Agents: How Much of What Passes Actually Tests the Bug? by Xiaonan Xu and Wenjing Wu. The method is called BSG-VA — buggy-state / candidate-state / gold-fix validation analysis — and the mechanic is simple enough that I'm mildly annoyed I hadn't thought of it. Every time the agent runs a validation command, the harness captures the exact working-tree state, extracts a test-only patch from it, and then replays that same command against three versions of the repo: the original buggy code (B), the agent's candidate state (S), and the developer's real fix (G).
Three replays give you a signature, and the signature tells you what the passing test actually proved. Fails on B, passes on S and G is gold-aligned bug-discriminating: it targets the defect and it survives someone else's implementation. Passes on all three is regression-only: it confirms nothing broke, and says nothing whatsoever about the reported bug. Fails on B and S but also fails on G is candidate-specific: it discriminates, but it's welded to the agent's particular patch. There are also misleading, flaky, diagnostic-negative and not-comparable buckets — seven mutually exclusive roles in total, assignable post hoc to any replayable trajectory that preserved its code states. Scale: 3,730 validation events across 643 rollouts on 110 tasks, drawn evenly from SWE-bench Verified and SWE-rebench (issues filed after the training cutoff, which matters here).
What makes this different from the reward-hacking work I've been reading all summer is the unit of analysis. SpecBench-style evaluation measures the gap between visible and held-out tests at the end of a run. This measures every individual validation event mid-trajectory and asks what it was worth at the moment the agent read it.
The key result#
Of the positive events where a three-way comparison was possible, 46.0% carried no bug-discriminating information at all. Regression-only accounts for the bulk of it — 1,141 events, 30.6% of everything captured — against 1,007 events (27.0%) that were genuinely gold-aligned. But the rollout-level number is the one that should bother you: in the baseline arm, with no feedback injected, 23.8% of rollouts closed with a submitted patch whose entire positive evidence base was non-discriminating. Nearly a quarter of the time, the agent finished, reported success, and every green check it was holding would have gone green on the unfixed repo too. Then the authors tried to fix it, and this is where the paper earns trust. Three arms: baseline; an attention-matched static reminder that a passing check may cover only part of the issue; and bug-contrast feedback, which replays each validation event on the buggy code in real time and injects the outcome back to the agent. Bug-contrast feedback beat the reminder by 7.8 points on evidence-inadequate closure (p = 0.0029) and 7.4 points on bug-discriminating evidence (p = 0.011), with no detectable cost to repair success. And then they say the quiet part out loud: both estimates fall below their prespecified 10-point smallest effect size of interest, about a third of the gain traces to the reminder alone, and the B-replay content adds a detectable increment only under one model-and-scaffold combination. They pre-registered a bar and then reported that they hadn't cleared it.
Why it matters#
This changes what "done" should mean inside an agent loop. Most of us instrument agent runs on outcomes — did it resolve, how many tokens, how many turns — and almost nobody instruments the evidence the agent used to decide it was finished. BSG-VA is post hoc and needs only a replayable trajectory plus preserved code states, which means you can run something like it on your own traces this week if you're already snapshotting the working tree before validation commands. If you're not, that's the change to make. The cheapest useful version doesn't need the full seven-role taxonomy at all: at submission time, take the tests the agent is citing as proof, run them against the pre-change checkout, and if they pass there, the agent's evidence is worthless no matter how confident the summary reads. That's a five-line harness addition and it converts a class of silent failure into a visible one.
The intervention result is the more interesting design lesson, and it cut against my prior. I'd have bet that handing the agent the B-replay outcome — hard, concrete, non-negotiable information — would comfortably dominate a generic nudge. It didn't, by much. Roughly two-thirds of the effect was the reminder; the specific evidence added a modest increment in one configuration. That's consistent with something I keep running into: a lot of agent "reasoning" failures are really attention failures, and the fix is a well-placed interrupt rather than richer context. But it also argues against solving this in the prompt at all. If prompting only buys you single-digit points on a behavior this basic, the honest move is to take it out of the model's hands — make the B-replay a gate in the harness instead of a message to the agent. Don't tell it the test doesn't discriminate; refuse the submission. That's the same place the skill-compilation and evidence-conditioning work keeps landing: deterministic steps belong in code, not in prose addressed to a model.
The caveats#
The gold fix as reference standard is noisy. Incomplete gold patches and genuinely multiple valid solutions both distort role assignment, and the authors flag it themselves.
Both models come from a single provider and family, and the task population is Python-ecosystem benchmarks only. The exploratory replications ran on a shared subset with few repetitions per arm and correspondingly wide intervals.
The confirmatory effects are real but small, and below the effect size the authors declared in advance. Treat bug-contrast feedback as promising, not proven.
Events before the first production edit are excluded, so this measures the agent's closing argument rather than its whole investigation. Reasonable — exploratory commands never claimed evidential weight — but it means the prevalence numbers describe a specific slice, not all tool use.
The takeaway#
What I'm filing away is the three-way replay itself, decoupled from the paper's intervention. B/S/G is a cheap, general test for whether any piece of automated evidence discriminates the thing it claims to, and it generalises well past program repair — the same question applies to any agent that hands you a green check as proof. One change to my own loops this week: before a repair agent is allowed to say "fixed," run its cited tests against the pre-change checkout. If they pass there, the run isn't done. It's unverified, and it should say so.