Humans reviewing alone catch about half of the bugs in a pull request. Add the best review agent and that rises to 82.1%. That single pairing number, from a new benchmark paper on code review agents, is a better argument for how to use these tools than the headline recall figure.
The paper, CRBugBench, takes bugs mined from repository history, pairs each with the PR that introduced it, and asks whether a review agent flags it in its top-k comments. There are 1,000 tasks across 98 open-source repos in five languages, plus 800 tasks from two proprietary monorepos at one large company. The title asks whether a software factory should be "lights-off", meaning no human in the loop. The data answers: mostly no.
The best agent is good, and the trend is steep#
Claude Opus 5 running in Copilot CLI catches 71.9% of bugs within its top five comments, averaged over two tracks (bugs caught pre-merge and bugs found post-merge). Opus 4.6 scored 47.7% on the same measure, so that is a 24-point jump in five months of model releases. Top-1 recall for Opus 5 is only 48.3% on the pre-merge track and 36.2% on the post-merge one. Top-10 reaches 86.0% and 68.6%.
Opus 5: 79.6 pre-merge / 64.2 post-merge, top-5
GPT-5.6-sol: 65.6 / 49.0
Sonnet 5: 64.7 / 45.2
Haiku 4.5: 43.0 / 23.0
The cost side is useful for anyone budgeting a review bot. Opus 5 runs about $2.97 per task and 9.8 minutes of latency. GPT-5.6-luna gets roughly 78% of Opus 5's performance at about 21% of the cost. If you review every PR, that ratio matters more than the leaderboard position.
Agents and humans miss different bugs#
This is the part I would build around. Humans were better at maintainability issues (odds ratio 2.70) and worse at severe bugs (0.45) and reliability issues (0.59). The agent is the opposite shape. It is better at the things a tired reviewer skips, and worse at the things a person with context judges almost by feel. A 50% human baseline rising to 82.1% with the agent in the loop is a complementarity result, not a replacement result.
The authors land in the same place. They support lights-off operation only for "narrowly scoped, low-consequence PRs" and say that for consequential changes agents are "better treated as scalable peers." I would put that sentence in the README of any auto-review setup.
A 100k-file monorepo took away up to a quarter of the recall#
On the proprietary repos, which average over 100k source files against about 3k for the open-source ones, performance fell by up to 25.5% relative to GitHub. Only 22.8% of the industry bugs were visible in the diff itself, versus 46.1% on GitHub. Diff-visible bugs are 2.6 times more likely to be caught (odds ratio 2.621), and visibility explains only about 20% of the gap, so the rest is project context the agent does not have.
Larger diffs hurt too (odds ratio 0.591 per the paper's regression), as did bug-fixing PRs (0.521) and usability bugs (0.454). Repository size and language had no significant effect on their own. So the lever is not "use a bigger repo-aware model", it is what you hand the reviewer: conventions, architecture notes, the callers of the changed function. The authors list supplying repository knowledge as one of their five recommendations.
More agents did not help the way more samples did#
On GPT-5.6-sol, base top-5 pre-merge recall was 65.6. Self-refine gave 67.1. An ensemble of three gave 68.0 and five gave 70.4, at roughly 3.68 times the cost. A mixed-model ensemble reached only 66.8, and a sub-agent setup actually dropped to 61.9. If you are tempted to split review into specialist sub-agents, this is a data point against it, at least for this task and this model.
Prompt changes were cheaper. A prompt that pushes the agent to explore more added 4 to 6 points for older models but only 0.3 for Opus 5, so I would retest any review prompt tuning each time the model underneath changes.
Where the review run actually dies#
The trajectory analysis gives a failure funnel: 95.6% of runs inspect the defective file, 80.2% produce some finding about it, and 56.2% actually hit the bug. Opus 5 writes a finding for the defect file 91.9% of the time, but only 75% of those diagnoses are correct. The agents spend about 75% of their turns reading and searching and about 8% executing code. Reading the right file is not the problem. Reasoning about it is.
That suggests the next gain is in giving reviewers a way to run something, a test or a repro, before commenting. That is my inference, not something this paper tests.
What the benchmark cannot tell you#
It measures recall of known bugs only. There is no precision or false-positive rate, which is the number that decides whether your team mutes the bot after a week. The benchmark favours mature, popular repos with reproducible builds, the regressions are associational, and the replication package for the industry data is not public. Bug validity was checked on 40 sampled bugs, of which 37 were valid, a small sample. Agent behaviour is stochastic too.
Still, I trust the direction. I would run a review agent on every PR as a second reader, scale its effort to diff size, feed it repo conventions, and keep a human on anything severe. My open question is the one the paper leaves open: at what false-positive rate does that 82.1% stop being worth the noise?