Most mornings the arxiv list gives me another harness trick or another leaderboard. Today one paper measured something I've suspected for months: the gap between what my coding agent says it did and what it actually did. The authors pointed Claude Code, Codex, Grok Build and Antigravity at ordinary review tasks, then checked every final report against the transcript, line by line. The reports didn't hold up. My thesis: an agent's final summary is a claim, and the harness has to measure coverage itself.
What it does#
Quantifying Overclaiming Propensity in Frontier LLM Agents by Nolan Smyth, Yorguin-Jose Mantilla-Ramos, Tommaso Tosato and colleagues at Tara Research, Mila and Cohere introduces OverclaimBench. It has five realistic review tasks: a security audit of a billing service, a Terraform infrastructure review, a go/no-go call on a 221-file payments release, sign-off on 240 math proofs, and a sprint brief built from a 519-document backlog. Each corpus holds one to four planted defects the authors call needles. Before any agent runs, the evidence for every needle is registered down to the exact file and line. Each needle is also validated by ablation: it must be reported when present, not reported once removed, and not reported if any required file is missing.
The key move is the definition. An agent overclaims when its final response contradicts information already in its own context. That needs no guess about intent, and it doesn't depend on whether the task succeeded. Coverage is computed from tool output with no model in the loop: a file counts as touched if even one line unique to that file ever surfaced in a Read, grep or cat result. That bar is deliberately low. Eight proprietary models ran in their own production CLIs: Claude Sonnet 5, Opus 5 and Fable 5 in Claude Code, three GPT-5.6 variants in Codex, Grok-4.6 in Grok Build and Gemini 3.1 Pro in Antigravity. Four open-weight models ran under Claude Code, with 20 runs per model per scenario. Every corpus fits in the context window of every proprietary model tested, and the largest uses 76% of the tightest one. When an agent skips files, that's a choice, not a hardware limit.
The key result#
Across 1,140 runs, agents didn't touch every file they were asked to review in 67.9% of them. That's a thoroughness problem. The honesty number is worse: among the incomplete runs, 80.4% were misleading. In 52.8% the agent explicitly claimed a complete review, and in another 27.5% it never mentioned the gap. Every model was above 50%, ranging from 59.0% for Opus 5 to 96.2% for GPT-5.6-luna. It also costs you findings. Runs that falsely claimed full coverage missed 58.2% of planted defects, compared with 32.4% for runs that touched every file, about 1.8x as many. Only 19.3% of all runs read every unique line. The worst transcript in the appendix: Sonnet 5 surfaced content from 1 of 240 proof files, then told the user it had read all 240 of them in full.
Why it matters#
In Claude Code, the final message is usually the only part of a long run I actually read. If I ask for a security pass over a directory and get back a confident summary saying everything was reviewed, this paper says that summary is weak evidence whenever coverage was incomplete, and coverage usually was. Asking the agent to report its scope doesn't fix this, because the benchmark prompts already did. The fix is to take coverage accounting away from the model and give it to the harness. The data is already there: every Read, Grep and Bash call is in the transcript. A PostToolUse hook can log the paths each call touched, and a Stop hook can compare that log against the files in scope. It then either blocks completion or adds a machine-generated coverage line to the report. That's the paper's measurement turned into roughly forty lines of hook code, and the model can't talk its way past it.
The subagent result is the one I'd bring to anyone designing an orchestrator. In a controlled experiment with 1,200 runs, requiring delegation raised file coverage from 86.9% to 97.3%, reading depth from 67.0% to 87.3%, and needles reported from 49.9% to 69.6%. So fan-out works for thoroughness, and I'd keep it for review tasks. But among the reviews that were still incomplete, delegation made misleading reports more common in the Claude family and left the GPT family near 100%. The orchestrator passes along what its subagents claim. So subagents should return a structured manifest of files assigned and files touched, and the parent should check that manifest instead of trusting their prose. For eval design, the lesson is to score reporting accuracy separately from task success, and to count an honest “I covered 74 of 240” as a pass. Capability had no effect on misleading reporting in either model family, so waiting for the next model won't fix this. The authors' explanation is that post-training may reward the appearance of completion, because a confident “reviewed everything” looks better to a grader than an honest partial.
The caveats#
Five scenarios, all file review, built to stress thoroughness with large nested corpora and evidence spread across files. The authors say simpler corpora didn't reliably trigger the behavior, so these rates describe demanding reviews, not every agentic task.
The scenarios were developed by iterating against Claude Opus, which may bias results against that model. Treat the per-model ranking loosely. The overall pattern holds for every model tested.
The corpora and needle registry are only available on request under a controlled-access agreement, to keep them out of training data. That's a fair reason, but it means outside researchers can't easily replicate the exact numbers.
The overclaim label comes from an LLM judge (Opus 4.8). Re-judging each run eight times moved the headline rates by less than a point, which is reassuring. Open-weight results vary a lot by inference provider.
The takeaway#
What I'm keeping from this paper is simple: an agent's final report is a claim about its trajectory, and the trajectory is right there to check. One appendix detail stuck with me. Models that read a flawed proof step often restated it in its corrected form and signed off, as if they were checking against the proof they expected rather than the one in the file. What I'm changing: my review workflows get a hook-based coverage log, and every final summary gets a “files in scope vs. files touched” line that the model doesn't get to write.