The arxiv cs.SE list is mostly new benchmarks these days, so a paper that runs no models at all stands out. This one takes 254 already-published SWE-bench submissions and asks a narrow statistical question: do the leaderboard's own verdicts support reading it as a ranking? At the top, the answer is no — and the variable that does move the numbers turns out not to be the model. My thesis: if you are choosing a coding agent by leaderboard rank, you are reading noise that your own harness dwarfs.
What it does#
Coding Agents Have Converged: Why the SWE-bench Leaderboard Can No Longer Order Its Top Entries, and What to Measure Instead, posted yesterday by Fengshuo Liu, Ying Liu, Ruize Sun, Lie Luo and Siyuan Guo, is a re-analysis rather than an experiment. The authors pull per-instance verdicts from 254 public submissions across four SWE-bench splits — Verified (134 submissions, 500 instances), Lite (84 / 299), Test (24 / 2,294) and Multimodal (12 / 301) — and never run a model themselves. Everything comes out of the pass/fail grids that submitters already published alongside their scores.
What separates it from the usual benchmark critique is what it declines to argue about. Most SWE-bench criticism attacks the tasks: contamination, underspecified issues, tests that accept the wrong patch. This paper accepts every verdict at face value and asks only whether the ranking is identifiable given those verdicts. It does that through three lenses: an overlap profile built on a nesting coefficient (cov(A,B) = |A∩B| / |B|, read against the score-implied baseline |A| / n), exact paired McNemar tests with Holm correction over adjacent leaderboard pairs, and a model × scaffold cell-mean analysis across the submissions that can be placed in both dimensions.
The key result#
On Verified, exact paired McNemar separates none of the 29 adjacent pairs in the top thirty at α = 0.05. The leading two entries each resolve exactly 396 of 500 instances. The top ten share 285 successes and 51 failures outright, leaving only 164 instances that distinguish any of them at all — and those solution sets are strongly nested, median 0.935 against a 0.774 baseline, meaning the frontier is not solving different problems but the same ones. Then comes the number that actually changes behaviour: holding the model fixed and swapping scaffolds (claude-3-5-sonnet across nine harnesses) produces a 29.8-point range — 168 to 317 instances resolved, 33.6% to 63.4% — against an 8.8-point spread across the entire top thirty. The harness moves the score more than three times as far as the whole frontier spans. Importantly, this is not just "every test is underpowered": on the 2,294-instance Test split the same procedure separates 14 of 23 adjacent pairs, which is the control that makes the Verified null credible.
Why it matters#
If a fixed model swings 29.8 points on scaffold alone while the top thirty models span 8.8, then retrieval strategy, edit-apply loop, test-execution feedback, context budget and retry policy are the dominant term in your system's performance, and model choice is close to a rounding error beside them. That is an allocation argument before it is anything else: an engineer-month spent on your harness has a larger expected payoff than one spent migrating models. It is also a reporting argument. A SWE-bench figure with no scaffold named is uninterpretable, and the paper's recommendation to publish model-scaffold provenance is the cheapest fix it proposes. That applies to internal numbers too — when I write down "we hit 62% on our suite," that line needs the harness version beside the model name or it is not even comparable to itself next month.
The nesting result quietly kills a common intuition. If frontier successes are 93.5% nested, ensembling the top three systems buys far less than their aggregate scores imply, because they fail on the same instances. The signal lives in the 164 discriminating instances, not the 285 everybody already solves. The paper prices this out: separating adjacent top-ten pairs at the observed discordance rates needs a median 52× multiplier, roughly 26,000 instances of similar character. But curating instances where discordance is imbalanced (a 0.2 rate favouring one system) brings that down to about 900. That is the actionable version — do not scale your golden set, sharpen it — paired with the methodological half: run paired tests on per-instance verdicts instead of comparing aggregate pass rates. Your fifty-task internal suite almost certainly cannot detect the differences you are making roadmap decisions on.
The caveats#
The model × scaffold analysis is observational, and 54% of submissions cannot be placed in a cell at all. Six of nine cell-mean interaction tests survive Holm correction, but scaffold authors also choose which model to pair with, so this is association, not an identified causal scaffold effect.
Non-rejection is not equivalence. "Indistinguishable on 500 instances" does not mean two systems are the same, and to the authors' credit they say so rather than overclaiming a tie.
One run per submission, so execution variability is excluded entirely. The real noise floor is wider than what is reported here.
Contamination is visible but unresolved — a 76% versus 53% file-identification accuracy split — though age-based leakage tests came back null, so it is flagged rather than explained.
This is a SWE-bench Verified result. Whether the same convergence has set in on other agentic benchmarks is left as future work.
The takeaway#
What I am filing away is less "SWE-bench is broken" than "SWE-bench Verified has run out of resolution at the top, and the leaderboard's own published artifacts are enough to prove it." Two things change for me after reading this. Every eval number I report internally now carries the scaffold and its version next to the model name, because the paper makes it obvious which of those two I was actually measuring. And I am rebuilding our internal suite around discriminating tasks and paired per-instance tests rather than a headline pass rate — a two-point gap on a fifty-task set was never telling me what I assumed it was.