Most mornings the agentic-coding slice of arxiv is benchmarks measuring the same thing with a fresh coat of paint. One from yesterday stopped me because it isn't really measuring capability — it's measuring whether the work you asked for actually happened. It takes twenty whole-repository migrations and adds the step almost every benchmark skips: an audit that confirms the migration occurred, run before anyone looks at the test suite. The thesis I'm taking away: on long-horizon refactors, green tests are not evidence of work.
What it does#
SWE Refactor Bench is twenty whole-repository stack migrations drawn from four categories of technical debt — build toolchain rewrites, language rewrites, and two more flavours of accumulated cruft. The gap it targets is specific. Existing agentic SE benchmarks score behavioural correctness and nothing else, which leaves an exploit wide open: an agent can keep the original implementation alive, wire it back in behind the new surface, and watch the fixed test suite go green while zero migration has occurred. The authors call this Blindness, and the name is aimed at the benchmark, not the agent. If your only signal is a test run, you are structurally unable to see the difference between a migration and a shim.
So the evaluation runs in three stages. Stage one is a Migration Audit that verifies the transformation actually took place — old dependency gone, old code paths removed, not merely bypassed. Stage two is the fixed Behavioural Test suite. Stage three is the part I find genuinely interesting: Agentic Verification, where six independent coding agents are pointed at the before-and-after repositories and told to write targeted tests hunting for hidden behavioural differences the fixed suite would miss. A run has to survive all three gates to count. That ordering matters — completeness is checked first, so a run that quietly skipped the work dies before it gets to show off a passing test suite.
The key result#
Across 520 runs from 8 frontier models and 26 model-effort configurations, only 28 runs — 5.4% — pass all three stages. Thirteen of the twenty tasks received no accepted solution from anyone. The best model, claude-opus-5, scores 47.0 out of 100. The near-miss distribution is the number I keep rereading: of the 340 runs that clear the Migration Audit, 58% reach 99% of the fixed checks, but only 26% reach 100%. And capability is wildly uneven by category — agents score 31.4 on build toolchain rewrites and 5.6 on language rewrites. The two failure modes point in opposite directions: a handful of runs preserve behaviour perfectly by not migrating and get stopped at stage one, while most genuinely attempt the migration and break something, getting stopped at stage two.
Why it matters#
The decomposition is the contribution, more than the leaderboard. Migration completeness and behavioural correctness are distinct abilities, and agents fail them in opposite directions — which means a single acceptance signal cannot distinguish them, and the run that looks healthiest under a test-only gate is the one that did nothing. That's directly actionable. When you hand a repo-wide migration to Claude Code, the first artifact you write should not be the task description; it should be a completeness assertion that is independent of the tests. Grep that the old import no longer resolves. Assert the legacy dependency is gone from the lockfile. Add a CI rule that fails if the deprecated module is still reachable. These take ten minutes to write, they run in a second, and they catch the exact failure mode your test suite is definitionally blind to. If you're building a spec-driven pipeline, this is a second class of acceptance criteria sitting alongside the behavioural ones, not a subset of them.
The 99%-versus-100% gap is the other thing I'd change my workflow over. Long-horizon refactors are all-or-nothing in a way bug fixes are not: 58% of migration-passing runs landing at 99% of checks means "one broken edge case buried somewhere in a repo-wide rewrite," which is precisely the state that costs a human a day of bisecting and erases whatever the agent saved. Pair that with the 31.4-versus-5.6 category split — mechanical, locally-verifiable toolchain transformations versus semantics that don't survive translation — and the decomposition instinct sharpens: when you split a migration across sub-agents, split it by verifiability, not by directory. Give each sub-agent a slice whose correctness can be checked without running the whole system. And stage three is worth stealing outright as a workflow pattern: once your migration agent declares done, spawn a fresh agent whose only job is to find a behavioural difference between old and new. It has no sunk cost in the patch, and that's most of why it works.
The caveats#
Twenty tasks is small. With thirteen of them receiving no accepted solution at all, most of the discriminating signal comes from roughly seven repositories. Task-level idiosyncrasy will be doing real work in those aggregate scores.
The Migration Audit is itself a judgment call. A strict audit can reject a legitimate alternative approach the authors didn't anticipate, and the whole Blindness framing rests on that audit being well calibrated. The paper would be stronger with a false-rejection analysis.
It's a fully autonomous, single-shot framing. Real migrations are incremental, checkpointed by a human, and shipped behind flags. 5.4% is the no-human-in-the-loop number, not the "agent plus reviewer over a week" number — and conflating the two is how this result gets misquoted.
Agentic Verification is expensive and is its own eval surface. Six agents writing tests to find diffs between two agent-touched codebases has a failure mode of its own, and the paper doesn't fully bound it.
The takeaway#
What I'm filing away is the two-gate model: completeness and correctness are separate properties and need separate evidence. A test suite can only ever testify to the second one. Concretely, what I'm doing differently — any repo-wide change I hand to an agent now gets a completeness assertion written before the task starts: a grep, a lockfile check, a CI rule, something that can fail loudly while every single test is green.