Most of this morning's agentic-coding list was benchmarks. This one is n=1 — a single case study, published by a company that sells the agent it evaluates, with a competing-interests declaration at the bottom. I nearly scrolled past it. Then I opened the tables. The headline is that an agent landed a refactor no human ever reviewed; the tables carry something more useful than that, which is evidence that the way most of us decide an agent verify loop is finished is simply wrong.
What it does#
Specification-first convergence with an AI coding agent (Joël Abenhaïm, AI Sovereign Labs) instruments one refactor end to end. The system is a 717,725-line TypeScript production application across 3,648 files — a VS Code extension that is itself an AI coding agent, closed source, written and maintained by the paper's author. The task was to dismantle a core lifetime invariant: the guarantee that a UI panel stays open for the entire duration of an AI request. The target behaviour was that a streaming generation survives its panel being closed and, on reopening, reattaches to the same live stream with no loss or duplication of tokens.
That task deliberately does not fit SWE-bench's shape. SWE-bench assumes an oracle — the correct behaviour is already encoded in a held-out test suite, and the agent hunts for a patch that passes it. Here the target behaviour did not exist before the change, so nothing could encode it. Compare the Bun rewrite the paper cites, 535k lines of Zig ported to Rust in eleven days by up to 64 parallel agents: that shipped because a million-assertion test suite and the Rust compiler were both oracles independent of the new implementation. Strip the oracle out and you need some other fixed thing to hold the code to. The protocol is five phases, each run as a separate session: ideate, specify, refine ×14, code, verify ×17. Refinement re-audits the specification against the real source. The spec then freezes. Code is generated against the frozen spec and applied without inspection. Verification re-audits the generated code against that same frozen document. The model is ChatGPT 5.6 Sol at max reasoning, under a harness carrying a roughly 250,000-character system prompt.
The key result#
Across the 31 audit passes, 201 defects, ambiguities and architectural deviations were identified and corrected before any human ran the code — roughly 85 in the specification, 116 in the generated code. The whole phase executed end to end without the program being run once; the first manual execution came after the seventeenth verification cycle, and the specified behaviour was there. The commit: 189 files, 23,663 insertions, 11,850 deletions. Three days, USD 2,430 of inference. But the number that actually changed my mind isn't in the abstract — it's the findings-per-cycle column of Table 3: 10, 12, 9, 21, 2, 7, 8, 13, 6, 3, 3, 4, 4, 4, 10, 0, 0.
Why it matters#
Read that sequence again, because it is not a decay curve. The peak is cycle 4, not cycle 1 — the audit got better at finding things as the code settled. Cycles 10 through 14 sit flat at three or four findings and look exactly like convergence; a reasonable engineer watching that plateau calls it done and ships. Cycle 15 then finds ten. The author's stopping rule — keep going until two consecutive passes return zero — is the only rule in the space that survives this data. "Findings are trending down," "findings are small now," and "we've done N passes" all fail on this run, and the failure costs you ten architectural deviations in production. Refinement shows the same shape: cycle 9 found about five, cycle 10 found eleven. So if you run any verify-loop pattern — a review sub-agent, a custom /verify command, a second Claude Code session auditing the diff — log the per-pass finding count and stop on consecutive zeros. A low, flat, non-zero plateau is not a convergence signal. It is the most misleading state the loop has.
The second thing worth stealing is where the independence lives. Current loop-engineering guidance, Anthropic's included, says the reviewing agent should be a different agent with no memory of the change, on the theory that fresh context is less biased. This protocol doesn't do that. The checker is a fresh session of the same agent under the same prompt. What is separated is the referent: a document written and frozen before the code existed. That's a cheaper and, I think, sharper reading of why self-review fails — the problem was never that the reviewer shares weights with the author, it's that a model revising its own output has nothing external to compare against. A frozen spec supplies the external comparand without a second agent, a second harness, or a second bill. Two side effects are worth noting. Refinement grew the file scope from 110 to 160 purely by re-reading the source, so fifty files of dependency surfaced before implementation instead of during it. And when handed the frozen spec, the agent refused to implement it in one shot, citing the spec's own atomicity requirement and proposing a decomposition instead — the abstain behaviour a lot of recent work has been asking for, showing up here because the spec gave it something concrete to abstain on.
The caveats#
n=1, and the author states it plainly. One task, one codebase, one operator, no control condition. Nothing here gives a success rate or a distribution of outcomes.
The author's company designed and distributes the agent under test, ran the operation, and wrote the report. 1,500+ pages of raw session logs are published (in French) so the process can be inspected, but the reporting is not independent and the competing interest is real.
"No bug observed" means: first manual run, roughly thirty later sessions, and a pre-existing test suite that showed no regression. That is a usage window, not an absence proof.
The codebase is closed source and was built by the same operator under this same methodology across some 4,000 commits. Spec quality on a system you wrote yourself is close to a best case; nothing tells us how the loop behaves on someone else's legacy code. The author names an independent operator on a public repo as the obvious next step.
USD 2,430 buys 31 audit passes on one change. That price is defensible for dismantling a load-bearing invariant and absurd for routine work — this is a protocol for the changes you would otherwise call a rewrite, not a default setting.
The takeaway#
What I'm filing away is the stopping rule and the curve underneath it. The headline result — an agent landing 189 files with nobody reading the diff — is n=1 from an interested party, and I'd want an independent operator on a public repo before I lean on it in an argument. The finding sequence needs none of that credibility; it's raw instrumentation, and it says a verify loop's plateau lies to you. So my review loops now stop on two consecutive clean passes rather than on my judgement that the findings have gotten small, and I've started logging per-pass counts so I can see the plateau instead of feeling it.