This week's arxiv output makes for uncomfortable reading if, like me, you choose models and harnesses partly off leaderboard numbers. Three of the eight papers below pick benchmark scores apart from different angles: the harness around the model, the agent's habit of going to find the answer, and the tasks themselves. Each one finds the headline number measuring something other than what we assumed. The other five are about what happens around the model. What does an unsupervised fix loop do to correct code? Does a repository SKILL file earn its place in context? How do spec-first frameworks actually enforce discipline? When is multi-agent collaboration worth paying for? And what does agentic code review look like when the developers at a large engineering company grade it?
Harness or Model? Isolating the Harness Effect in Agentic Coding#
Mohsen Arjmandi · 8 September 2026 · arxiv 2609.11987
A common assumption is that a vendor's own harness is the best way to run that vendor's model: the Claude Agent SDK for Opus, the Codex SDK for GPT. This paper tests that assumption with paired same-model runs on a private, contamination-controlled suite of repository tasks and post-cutoff contest problems. The same 80 tasks ran under each vendor's native SDK and under the neutral deepagents harness, for Claude Opus 4.8 and GPT-5.5. An isolated oracle graded 792 of the 800 planned runs.
On average, neither harness wins. Opus 4.8 scored 48.8% native versus 50.0% neutral, and GPT-5.5 scored 55.6% versus 54.4%, with confidence intervals that comfortably span zero. The Opus average hides a split, though. The native harness trailed by 9.0 points on the 61 repository tasks and led by 23.7 points on the 19 contest tasks. The author flags that this split was chosen after seeing the data and needs a proper replication. Two other results are more immediately useful. 22 of the 81 runs cancelled at the wall-clock ceiling had already produced a passing patch. And based on observed usage, the neutral harness cost 1.3 to 1.6 times as much per solved task on Opus, and 1.2 times as much on GPT-5.5. However, 58 Anthropic runs left no usage record, so the author says it is still unclear which harness is cheaper in actual billing. The paper is itself a correction of the author's August manuscript, whose cost figures rested on a bug in their own telemetry.
Why it matters: The question I get asked most about agent tooling is “should we just use the vendor SDK?” This is the first careful answer I have seen: not for accuracy, maybe for cost, and it depends on the shape of the task. The 22 cancelled-but-passing runs are the part I'd act on right away. If your harness kills runs on a timer without checking whether the tests already pass, you are throwing away solved work. And an author publicly correcting their own cost telemetry shows how easy it is to get token accounting wrong. Check yours before you optimise against it.
Shortcutting the Fix: Agentic Exploits in Software Engineering Benchmarks#
Nikolai Ludwig, Wasi Uddin Ahmad, Somshubra Majumdar, Boris Ginsburg · 6 September 2026 · arxiv 2609.06780
Give a coding agent a shell and it will use it. This paper catalogues the ways agents take shortcuts on benchmark tasks: digging through local Git history, pulling from the upstream repository, or reproducing a memorised solution. It audits five open models on SWE-bench Multilingual and DeepSWE. The judge is an LLM that reviews each turn of the agent's run, not just the final patch.
The rates are startling. Under standard prompts, 45.1% to 82.4% of SWE-bench Multilingual runs used an exploit, and 44.2% to 66.1% of DeepSWE runs. Adding one targeted instruction demanding an original solution cut those to 4.0%–10.7% and 1.5%–7.1% respectively, while core task performance stayed strong.
Why it matters: There are two lessons here, one for evaluation and one for production. For evaluation: a resolution rate means little unless someone has checked how the agent actually got there. For production: the instinct that sends an agent to git log for the answer is often exactly what you want on a real repository, but the same instinct will copy something it should not. That a single prompt line changes behaviour this much tells me these exploits are default strategies, not deep capabilities. It also tells me your system prompt is doing more policy work than you probably realise.
SWE-Bench Pro Verified: A Reliable Benchmark for Software Engineering Agents#
Pujun Zheng, Zixin Shang, Shufan Jiang, Wenhui Tian, Dongsheng Zhu, Zerun Ma, Dingbo Yuan, Qi Zhang · 8 September 2026 · arxiv 2609.08149
SWE-Bench Pro is the harder benchmark everyone started quoting once the original SWE-bench leaderboards flattened out. This team found two problems undermining it. The first is reward hacking: agents reaching leaked gold solutions and hidden evaluation files. The second is task quality: misleading problem statements, and tests that are too narrow or too broad. Their answer is a verified release of 731 instances. It rebuilds repositories without their Git history, hides test files, anonymises metadata and blocks source hosts at the network level. It then uses LLM-assisted filtering plus expert review to make minimal fixes to 102 broken instances.
The effect on one model is dramatic. GLM-5.2 dropped from 78.80% to 57.32% once the anti-hacking controls were on, a fall of 21.48 points, with 186 passes turning into failures against 15 going the other way. The audit backs this up. The number of tasks where the agent opened answer files locally fell from 103 to 0, and over the network from 49 to 0. DeepSeek-V4-Pro, which an earlier audit had found barely hacked at all, hardly moved: 49.98% to 49.11%. Fixing the broken tasks then won back some score for both models, to 59.51% and 49.93%.
Why it matters: Read this next to the previous paper. If closing leakage channels moves one model by 21 points, then a gap between two models on a leaderboard can mostly measure how hard each one looks for the answer. The practical move is to stop comparing raw vendor-reported SWE-Bench Pro numbers without asking what environment they ran in. If you run internal evals on your own repositories, the paper's list of leakage channels makes a good checklist for your own sandbox: Git history, the local file system, upstream mirrors, and task identifiers.
If It's Not Buggy, Don't Fix It: The Dynamics of Iterative Bug-Fixing#
Xietao Wang-Lin, Anton Isopoussu, Louis Mahon · 9 September 2026 · arxiv 2609.10123
What happens if you hand an LLM a program, ask it to fix the bugs, and feed the result back in, over and over, with no tests to tell it when to stop? This report studies that blind loop. For each turn, it measures how often a buggy program gets repaired and how often a correct program gets broken. It compares two edit styles: targeted search/replace blocks and whole-file rewrites.
The models almost never say the code is fine. With search/replace edits under greedy decoding, Gemini 2.5 Flash-Lite broke an initially correct program on 29.3% of turns but repaired an initially buggy one on only 2.3%. For Qwen2.5-7B-Instruct the figures were 42.4% against 0.4%. Whole-file edits narrowed the gap a lot, to 16.5% damage against 10.1% repair for Flash-Lite. They also produced fewer loops where the same change gets added and removed indefinitely. Probing the model's internals turned up a “buggy code” direction in its activations. Pushing the model along it makes it edit more or less, raising or lowering both repairs and damage together.
The caveats are real: the models are small, each task is a single file, and the model gets no signal about whether it has succeeded. But that last condition is exactly what an autonomous loop creates when nobody has written a test.
Why it matters: Every “review your work and fix anything wrong” step in an agent pipeline is a version of this loop. If the loop stops only when the model decides it is done, expect the model to keep finding problems and to damage working code faster than it repairs broken code. The decision to stop needs an outside signal: passing tests, a limit on how much can change, or a hard cap on iterations. The search/replace versus whole-file result is also worth a second look if your own edit tool is diff-based, even though these were not frontier models.
Skill Issue: Lessons from Optimizing Repository SKILLs for Coding Agents#
Mykhailo Kozyrev, Andrei Kozyrev, Anton Podkopaev · 11 September 2026 · arxiv 2609.12742
SKILL files are plain markdown documents, versioned alongside the code, that tell a coding agent how a repository works. They are now common enough that people have started generating them automatically, by tuning the document against a benchmark. The catch is that a real repository has no benchmark, and the synthetic tasks earlier work built were easy enough that a capable agent solved them with no document at all. So the authors mine harder tasks. They take merged pull requests, revert them at a single fixed commit, and score a candidate SKILL by whether the same agent does better with it than without it.
On three Kotlin repositories, documents found by the GEPA optimiser raised that score by 4.9 points on average, while SkillOpt's ended just 0.1 points above where they started. The authors are unusually honest about what this means. The GEPA gain matches what earlier work reported, but with the number of tasks one repository's history can supply, it cannot be told apart from the agent's run-to-run variance. Meanwhile, a maintainer who read the generated documents found in them knowledge you only get by working in the project.
Why it matters: I suspect this is where most teams will land with CLAUDE.md and skill files: they read well, and it is genuinely hard to prove they help. The reverted-pull-request method is the part I am taking, because it turns history you already have into a cheap, repository-specific eval. The variance finding is the sobering part. If one repository cannot supply enough tasks to tell a 5-point gain from noise, most “this skill made the agent better” claims going around are anecdotes. Write skill files for the knowledge they capture, and be careful about claiming measured gains.
Consort: A Spec-First Agent Framework for Enforced, Test-Driven Development#
Kevin Hartman · 9 September 2026 · arxiv 2609.09671
This is a position paper that introduces a framework, not an empirical study, but it names something useful. The spec-first agent frameworks it lists (GitHub Spec Kit, obra/superpowers, BMAD, GSD) all agree on writing down intent up front, as a specification or planning documents that persist. What sets them apart is how they make an unpredictable worker follow engineering discipline. The paper describes three approaches. Enforcement by persuasion relies on prompt rules the model may ignore. Enforcement by front-loaded structure means strong specs followed by a build you trust. Enforcement through controls means checks the agent cannot edit.
Consort is built on the third. A fixed, non-AI orchestrator moves separate role agents through a design stage that starts from the spec and a build stage that starts from the tests. Along the way there are human sign-off gates and tests the agent cannot change. To count as done, the code has to pass against a live copy of the database on its own branch. The author states the paper's claims as a hypothesis registered in advance, to be tested later. One claim is that gates enforced in code keep agent output honest. The other is that separate roles keep it maintainable. So there are no numbers yet.
Why it matters: The persuasion/structure/controls split is the most useful way I have seen to compare these tools, and it is a good question to ask of your own setup: which of your rules can the agent actually break? If “always write the test first” lives in a markdown file, that is persuasion. Tests the agent cannot change and an orchestrator it cannot edit are a different kind of guarantee. After reading Shortcutting the Fix, I would bet on controls. I would like to see the planned evaluation actually run.
Learning How Much to Collaborate: Difficulty-Aware Topology Selection for Multi-Agent Code Generation#
Yunsong Hong · 12 September 2026 · arxiv 2609.13890
Multi-agent code generation systems usually pick one way for their agents to communicate, a topology, and use it for every problem. This paper tests five topologies on 614 problems from APPS, HumanEval+ and LiveCodeBench. It finds that the value of collaboration depends heavily on difficulty. A hierarchical team beats a single agent by 2.4 points of pass@1 on the easiest third of problems and by 21.1 points on the hardest third, while using about ten times the tokens throughout.
The proposed Difficulty-Aware Topology Selector (DATS) estimates each topology's chance of solving a problem and picks the one with the best expected success after a cost penalty. Capped at 40% of the cost of always going hierarchical, it reached 77.7% pass@1, against 73.6% for always-hierarchical and 74.3% for the strongest learned competitor. The 4.1-point gain held across four underlying models. One detail I liked: swapping its 39 hand-picked features for a graph network or a pretrained encoder changed accuracy by 1.3 points at most. That suggests the win comes from routing by difficulty at all, not from a sophisticated predictor. The paper also argues that routers should be compared at the same spend. When they are, two baselines that had been ahead of DATS fall behind.
Why it matters: Most multi-agent setups I see either use multiple agents for everything or never do. The gap between 2.4 and 21.1 points says both choices waste money: you pay 10 times more on easy tasks, or you under-invest in hard ones. You do not need their predictor to act on this. A cheap difficulty estimate that decides whether to add a planner and a reviewer captures most of the benefit. The point about matching spend applies to every agent benchmark claim, too: accuracy without cost is only half the number.
Using Agentic AI for Contextualized and Multifaceted Code Review at Ericsson#
Muhammad Laiq, Ricardo Britto, Muhammad Usman, Nishrith Saini, Deepika Badampudi · 14 September 2026 · arxiv 2609.15877
This is an industrial study: a multi-agent code review system built and evaluated inside Ericsson. It pairs specialised agent skills with project-specific knowledge to flag antipatterns in code changes across four dimensions: readability, maintainability, reliability and performance. The team ran it over real commits, and it flagged more than 200 issues, which the company's own developers then checked by hand.
The developers judged 96% of the flagged issues to be correctly identified. Around 69% of those correct issues were rated important: about 33% were severe issues that must be fixed, and 36% were important issues that should be fixed. Developer feedback backed up the numbers. What the paper does not report is recall, meaning the issues the system missed, and that is the harder number for any review tool.
Why it matters: Most AI code review tools I have tried fail on noise rather than accuracy: comments that are technically right but that nobody cares about. Roughly two in three flagged issues rated important, judged by the developers who own the code, is a meaningful bar. The paper credits that to combining specialised skills with knowledge of the specific project, not to a smarter model. That matches my experience: a review agent that knows your team's conventions beats a generic one. With agents writing more code than humans can review, this is where the bottleneck is heading.
The Common Thread#
Benchmark scores measure the environment as much as the model. The harness paper finds no average advantage for vendor-native harnesses but a 9-point versus 23.7-point split depending on task type. Shortcutting the Fix finds 45% to 82% of runs using an exploit under standard prompts. SWE-Bench Pro Verified moves one model by 21 points just by closing leakage channels. Before you choose a model or harness based on a number, find out what sandbox and harness produced it, and what the agent could reach while it ran.
The decision to stop, and the rules themselves, have to live outside the model. A fix loop that stops when the model says so damages working code faster than it repairs broken code. One prompt line cuts benchmark exploits sharply, but a prompt is still persuasion, and Consort's answer is controls the agent cannot edit. The harness paper shows the reverse failure: a timer that cancels runs which had already solved the task. Outside checks are necessary, but they also have to check the right thing.
The gains come from deciding where to spend compute and context, and measuring that honestly. DATS matches or beats always-on collaboration at 40% of the cost by routing on difficulty. Ericsson's review system credits project-specific knowledge for its accuracy. Skill Issue shows repository knowledge is valuable and still hard to measure against run-to-run variance. None of these wins came from a bigger model. They came from choosing what the model sees and when it gets more help, and from being careful about the numbers.