Something shifted in this week's arxiv output. For about two years the interesting question in agentic coding was whether the model could write the code. That question is largely settled, and the papers landing now have moved on to a less comfortable one: how do you tell an agent what “done” means, and how do you check that it got there? Seven papers from the last ten days, and almost all of them are arguing about specification and verification rather than generation. One measures the cost of that checking directly. Another shows that our favourite success metric — the tests pass — is quietly overstating how well agents actually do.
SWE-Gate: Passing Functional Tests Is Not Enough#
Xin He, Yanlin Wang, Mingwei Liu, Jiachi Chen, Hongyu Zhang, Guanbin Li · 3 September 2026 · arxiv 2609.04167
SWE-Gate is a repository-level benchmark that separates functional correctness from what the authors call review constraints — the acceptance conditions that show up in real pull request review comments rather than in tests. They mine those constraints from actual PR reviews, then synthesise repair instances around them: 303 instances across 75 open-source Python repositories, each shipping separate functional tests and constraint tests alongside a non-compliant patch and a gold patch.
The headline number is the gap. Across four LLM backends running on a common coding-agent scaffold, of the 644 repairs that passed the functional tests, 221 failed the review constraints. A bit over a third of apparently successful patches would have been sent back by a human reviewer.
Why it matters: This is the number I want stapled to every SWE-bench score I read. When an agent goes green, “green” means it satisfied the tests that happened to exist — not the unwritten rules about how this codebase handles errors, or which internal helper you are supposed to use instead of rolling your own. Those rules live in review comments, which is exactly where SWE-Gate went looking. A one-third slippage rate between “tests pass” and “acceptable” matches my experience of reviewing agent PRs uncomfortably well, and it explains why the review queue, not the generation step, is where agentic throughput actually dies.
The Reach of a Verification Tool Decides Its Value#
Achint Mehta · 28 August 2026 · arxiv 2608.28795
This is the best-designed experiment in the batch. Build a minimal coding agent whose tool list is the single controlled variable, then use it to implement 1,116 web applications across six models and eight tool configurations. A condition-blind human grades every application against a frozen rubric, and automatic probes stress-test the API-observable behaviour. The question: does widening an agent's ability to check its own work produce a matching improvement in what it ships?
Mostly no — but the first slice is nearly free. Without any tools, roughly one build in seven fails to launch at all, and a single boot probe removes nearly all of those failures at about 35 percent of a full shell's token cost, while giving the agent a full shell multiplies the no-tools cost by 2.35×. Screenshots help where mistakes are visible, such as element placement and interaction, though the gain over a shell is modest and does not survive correction for multiple comparisons. Where failure can only be measured rather than seen — keeping scrolling smooth over a 100,000-row list — screenshots add nothing at all.
Why it matters: I have been guilty of the give-the-agent-every-tool-and-hope approach. This reframes verification tooling as a set of narrow detectors rather than a general-purpose quality dial: a tool improves the artifact only where its reach covers the way the application actually fails. The practical order of operations is add the boot probe first, because one in seven is a shocking baseline and it is the cheapest large win on the board, then add detectors matched to your real failure modes rather than the impressive-looking ones. A screenshot tool feels like it ought to catch a performance regression. It cannot see one.
Requirements After the First Edit#
Bowen Jiang, Haowei Cheng, Yuhong Fu, Anne Koziolek, Jialong Li, Weixing Zhang · 2 September 2026 · arxiv 2609.03028
This one mines 3,553 SWE-chat sessions for requirements that arrive after the agent has already started editing, then links each arrival to a proxy for the damage it causes: deletion or replacement of lines the agent previously wrote. Curated benchmarks fix requirements before implementation by design, so this churn is invisible in almost every evaluation we rely on.
A requirement arrival is followed by roughly twice as much invalidation as matched non-requirement edits — robust to user-turn and net-deletion checks, though the authors are careful not to claim causality. Two secondary results are the ones that stayed with me. The burden shows no detectable decline over the course of a session. And in a controlled experiment, advance warning that a requirement was coming produced no detected effect on overwriting; delayed disclosure simply relocated the implementation work to after the reveal.
Why it matters: This is the requirements-engineering classic — stakeholders cannot articulate a constraint until part of the system exists to react to — and agents make it worse precisely because they reach “something exists” so fast. The lack of within-session decline is what stings, because it kills the comfortable intuition that you and the agent converge on the spec as you go. And advance warning not helping suggests you cannot prompt your way out of it. What is left is structural: make the cheap-to-throw-away artifact first. This is the strongest empirical case I have seen for spending a round on a plan or spec document before the agent touches code — not because planning is virtuous, but because plan text is cheap to invalidate and implemented code is not.
Spec-Driven Development for Agentic Software Engineering#
Jessica Diaz, Joaquin Gayoso, Andrea Cimminio, Jorge Perez · 31 August 2026 · arxiv 2609.00252
The conceptual companion to the paper above. It tries to give spec-driven development a proper foundation as the discipline that makes agentic software engineering work at team scale. It is built largely on gray literature — practitioner reports, talks, tooling, vision papers — because, as the authors say plainly, a peer-reviewed evidence base and a shared vocabulary do not yet exist.
Three contributions: a socio-technical model in which specifications act as the contract substrate between humans and agents; an operational characterisation of the harness that splits it into a technical harness around the agent and a methodological harness around the team; and a typology of five human-agent interaction patterns. The framing I keep returning to is the productivity paradox they open with — individual productivity rises while team throughput, review capacity and stability degrade.
Why it matters: I have been looking for language to describe why a personal agent setup scales beautifully and a team's does not, and technical harness versus methodological harness is it. Your CLAUDE.md, your hooks, your slash commands — that is the technical harness, and it is the part everyone optimises because it is the part you can commit. The methodological harness is who reviews what, which specs are binding, and how disagreements between agent output and house style get settled. Nobody ships that as a plugin. The paper is honest that this is a first step toward consensus rather than a validated theory, so take it as vocabulary rather than gospel — but the vocabulary alone earns the read.
A Phased Workflow for Operating LLM-Based Coding Agents#
Ante Kapetanovic, Tomislav Duricic, Andro Mercep, Emanuel Lacic (Infobip) · 31 August 2026 · arxiv 2608.30701
An experience report from Infobip's AI research team on how they actually structure work with coding agents. Four phases, with human effort front-loaded and delegation increasing as artifacts mature. Context management is treated as the central concern, addressed through four strategies applied at each phase against known failure modes.
Two observations drive the design. Upstream errors in research and planning compound across later phases, and correcting generated code tends to introduce bloat and fragility rather than converging on something clean. Together those justify front-loading human review. The authors also name two open problems honestly: there are no metrics for workflow effectiveness, and there is a gap between the formalised context-management components in the literature and the workflow-level patterns practitioners actually need.
Why it matters: Read this next to the requirements paper and you get the same conclusion from two directions — empirically, late spec changes cost about double; operationally, a team shipping this for a living concluded you have to front-load. The observation I found most useful is that patching generated code adds bloat and fragility. It matches a pattern I hit constantly: the third fix on a bad generation is almost always worse than regenerating from a corrected prompt, and yet the sunk-cost pull toward patching is strong. This is one team's experience without effectiveness metrics, and they say so themselves, but it is a well-articulated one.
Harness-of-Harness: Multi-Day Autonomous Development#
Haoyang Yan, Min-le Su, Hangfan Zhang, Zhanhao Li, Chen Zhang, Shao Zhang, Yang Chen, Lei Bai, Shuyue Hu · 1 September 2026 · arxiv 2609.01481
A meta-harness. HoH sits on top of existing coding-agent harnesses and organises their executions into iterative planning-coding-testing loops, aiming at continual improvement over a long horizon rather than one-shot task completion. The design choices are the interesting part: scope development into small verifiable increments, separate implementation-time testing from independent evaluation, progressively expose deliverables and role-specific skills, and constrain verifiable outputs rather than prescribing agent workflows.
On GameCraft-Bench, FrontierSWE and ProgramBench, with three harness-model pairs (Codex with GPT-5.5, OpenCode with DeepSeek-V4-Pro, Pi with MiniMax-M3), HoH beat the corresponding standalone harness in every case — an average relative gain of 52.25 percent and a maximum of 82.86 percent after three iterations. In a multi-day deployment running more than 70 iterations, it autonomously built a first-person shooter with a coherent storyline, implemented core mechanics, polished visuals and integrated audio.
Why it matters: “Constrain verifiable outputs rather than prescribing workflows” is the line worth putting on a sticky note. It is the verification-surface lesson from the other end: specify what must be checkable, then stay out of the way of how the agent gets there. A 52 percent average gain over the underlying harness, consistent across three different model-harness pairs, is a strong signal that a lot of current headroom sits in orchestration rather than in the model. I would temper it with the obvious caution — game-building benchmarks reward exactly the incremental, independently verifiable structure HoH imposes, and I would want to see this on gnarly brownfield work before assuming the number transfers.
The Maintenance and Co-evolution of Agent Plugins#
Ahmed Hereiz, Yingzhe Lyu, Hao Li, Bram Adams, Ahmed E. Hassan · 28 August 2026 · arxiv 2608.28497
The first serious empirical look at the Claude Code plugin ecosystem as a software ecosystem: 1,926 repositories hosting plugin marketplaces, 8,351 plugins and 77,773 commits across 2,018 marketplaces. The framing question is whether agent plugins — which deliver functionality through a mix of natural-language instruction files, scripts and configuration — are maintained artifacts that co-evolve, or one-off things people write once and never revisit.
Plugin-touching commit activity grew 8.8× over the six months following the October 2025 launch, and 61.3 percent of plugins target software engineering tasks. Development is far more feature-driven than normal open source: 39.6 percent feature commits against 17.2 percent in conventional OSS. Claude co-authors 34.9 percent of all commits. And the finding with real teeth — inside skills directories, natural-language instruction files and implementation scripts co-evolve at above-chance rates, with 78 percent of those co-changes functionally coupled. The authors call it a new class of maintenance dependency not observed in traditional software engineering.
Why it matters: That last result is genuinely new to me. We have all internalised that code and tests co-evolve, and we have tooling and review norms to match. Prose instructions and the scripts they describe are now coupled the same way, and we have nothing — no type checker, no test, nothing that fails when a SKILL.md drifts out of sync with the script it documents. Seventy-eight percent functional coupling means this is not incidental. If you maintain skills or plugins, review the instruction file and its scripts as a single unit, the way you would never merge a signature change without the call sites. The 39.6 percent feature-commit rate tells its own story too: this ecosystem is still in land-grab mode rather than maintenance mode, so a good share of what you install today are one-off artifacts, not maintained ones.
The Common Thread#
Generation is no longer the bottleneck — specification and verification are. Six of these seven papers are, at bottom, about the same gap: the agent will confidently produce something, and the hard part is expressing what correct means and then checking it. SWE-Gate quantifies the distance between passing tests and being acceptable. The verification-surface study quantifies what checking costs. The requirements paper quantifies what a late spec costs. None of them are about making the model better at writing code.
Cheap, narrow checks beat broad, expensive ones. A boot probe at 35 percent of shell cost removing nearly all launch failures; HoH's insistence on small verifiable increments; SWE-Gate's separate constraint tests. The pattern is targeted detectors matched to real failure modes, not maximal tooling. The corollary is unglamorous: you have to know how your system actually fails before you can choose tools well.
The harness is becoming a maintained artifact with its own decay. The plugin study makes it concrete — instruction files and scripts co-evolving at 78 percent functional coupling, with no tooling to enforce it. The spec-driven development paper splits the harness into technical and methodological halves and points out we only maintain the first. Your CLAUDE.md, your skills, your team's review norms: that is load-bearing source now, and most of us are still treating it as configuration you write once.