Five papers landed this week that, read together, keep circling the same uncomfortable question: now that coding agents are being handed real production work and multi-hour tasks, how do we actually know they are any good — and can they check their own work? The short answer running through all of them is that verification has to come from somewhere other than the model doing the coding. But there is a more hopeful thread too: the field is quietly moving past “did the patch pass” toward measuring how an agent got there and what it cost me to babysit it.
RigorBench: Benchmarking Engineering Process Discipline in Autonomous AI Coding Agents#
Most benchmarks grade the patch. RigorBench (Madiraju & Madiraju) grades the process — the argument being that an agent that stumbles into a correct answer through reckless trial-and-error is a bigger production liability than one that plans, verifies, and knows when to stop. It scores agents across five dimensions — Planning Fidelity, Verification Coverage, Recovery Efficiency, Abstention Quality, and Atomic Transition Integrity — rolled into a single RigorScore, over 30 tasks in categories whose names tell you exactly what they probe: Plan-Then-Build, Verify-Or-Die, Doom Loop Gauntlet, Know When to Fold, and Don’t Break the Build.
What makes this one land for me is that they evaluated actual agent harnesses — Agent-Skills, Superpowers, Agent-Rigor — not just raw models. And the numbers argue that discipline is not merely aesthetic: structured process discipline lifted process-quality scores by an average of 41%, and, crucially, dragged downstream outcome correctness up 17% along with it. Good process is not a tax on getting the right answer; it correlates with getting it.
Why it matters: If you build harnesses, this is a benchmark that finally rewards the scaffolding work — planning, verification loops, graceful recovery, abstention — instead of treating it as invisible. It is evidence you can point to that the harness is the product, not a thin wrapper around the model.
SWE-Together: Evaluating Coding Agents in Interactive User Sessions#
Almost every coding benchmark is a one-shot: here is an issue, produce a patch, grade it. That is not how anyone actually uses these tools. SWE-Together (Wu et al.) rebuilds evaluation around the multi-turn reality by reconstructing real user–agent sessions — 109 repository-level tasks distilled from 11,260 recorded sessions, filtered to cases where the repo state could be recovered and the user’s intent was clearly documented. An LLM-based user simulator then replays consistent feedback across different agents, so you can compare them on the same conversation.
The headline result is intuitive but worth having measured: stronger agents do not just reach higher final success rates, they get there with fewer corrective turns. That second number is the one I care about — it is the babysitting cost, the amount of human steering a tool demands to land the fix.
Why it matters: Final pass rate is half the story. The metric that predicts whether people will actually keep an agent in their loop is how many times they have to stop and correct it. Evaluating on corrective turns, not just terminal correctness, is the right unit for anything that ships to real users.
When AI Reviews Its Own Code: Recursive Self-Training Collapse in Code LLMs#
Here is the loop that keeps me up: AI-generated code gets committed, the repo later becomes training data, and now the model is learning from its own past output at repository scale. Song, Cai, and Zhao study what happens when that loop closes with different gatekeepers. No review at all collapses fastest. A human gate slows the decay but does not stop it. And the seductive option — letting the model gate its own code — looks fine at first, then slides into what they call a rubber-stamp regime: acceptance scores keep climbing while benchmark correctness quietly falls.
The punchline is a clean, almost inconvenient result: stable recursive training of code LLMs requires exogenous verification, not model-coupled self-review. A model grading its own work eventually optimizes for its own approval rather than for correctness — the two decouple, and confidence becomes an actively misleading signal.
Why it matters: Every “have the agent review its own diff before shipping” pattern lives on borrowed time. Self-assessment is not a substitute for an external check — tests, a separate verifier, a human — and rising self-confidence is exactly the wrong thing to trust. This is the theoretical backbone under a lot of the recent reward-hacking and safety work.
Dockerless: Environment-Free Program Verifier for Coding Agents#
If the last paper says you need external verification, the obvious objection is cost — spinning up a Docker container per repository to actually run the tests is expensive and does not scale to training pipelines. Dockerless (Zeng et al.) sidesteps that entirely: it judges whether a patch is correct without executing it, using agentic exploration of the repository to gather evidence instead of a live environment.
And it works surprisingly well for something that never runs the code. It beats the strongest open-source verifier by 14.3 AUC points and reaches a 62.0% resolve rate on SWE-bench Verified, 50.0% on Multilingual, and 35.2% on Pro — improvements of 2.4, 8.7, and 2.9 points over the Qwen3.5-9B baseline, matching environment-based post-training approaches without the environments. That opens the door to using verification as a reward signal in RL and as a trajectory filter in SFT, at a scale that per-repo Docker setup makes impractical.
Why it matters: This is the practical complement to the self-review-collapse result. Exogenous verification is necessary but has been expensive; making it environment-free and cheap is what lets you actually bake it into training and inner loops instead of just recommending it in the abstract.
The Shift to Agentic AI: Evidence from Codex#
The other four papers are about how to evaluate and verify agents; this one is about how fast the ground is moving under that question. Johnston, Holtz, and colleagues analyze Codex usage and find the active user base grew more than fivefold in the first half of 2026 — with the fastest growth outside the developer community. Inside OpenAI, adoption is nearly universal and has largely displaced ChatGPT for business tasks.
The sophistication signals are the part that matters for tool builders. Over 10% of users now run three or more concurrent agents in a given week, 26.6% use “skills” to share complex workflow instructions, and there has been a nearly tenfold jump in users submitting tasks that would take a human expert 8+ hours. Output scaled accordingly — legal professionals generated 13x more monthly tokens than seven months earlier, researchers over 50x.
Why it matters: People are already running fleets of agents on multi-hour tasks, reusing skills, and delegating work no one is going to hand-review line by line. That is exactly the regime where the first four papers stop being academic: process discipline, interactive-turn cost, and cheap external verification are what stand between “fivefold growth” and “fivefold growth in unverified code.”
The Common Thread#
Verification has to come from outside the model. Self-review collapses into rubber-stamping (Recursive Self-Training), so the fix is exogenous — and Dockerless shows that external check can be made cheap enough to actually deploy in training and inner loops.
Evaluation is growing up. RigorBench grades how an agent works, SWE-Together grades what it costs you to collaborate with it — both moving past the single terminal pass/fail that no longer separates good agents from lucky ones.
The adoption curve is outrunning the rigor. The Codex data shows fleets of agents on multi-hour, low-oversight tasks; that scale is precisely what makes disciplined process and cheap external verification urgent rather than nice-to-have.