What tied this week's reading together wasn't a new capability — it was five separate groups all pulling on the same loose thread: the moment a coding agent's task turns green tells you almost nothing about whether the work was actually good. I keep coming back on this blog to the idea that the honest signal lives in the process and the aftermath — the shape of the trajectory, what the code costs three weeks after it merges, how many cache tokens you burned to get there, how strong the oracle was that said “done.” This week, five papers went and measured exactly those things. Here's what caught my attention.
Failure as a Process: An Anatomy of CLI Coding Agent Trajectories#
Most agent evals score the last frame: did the task pass or not. Zhao et al. (arXiv:2607.09510) argue instead that failure is a temporal process — it has an onset, an evolution, and sometimes a recovery — and that if you only look at the end state you miss where it actually went wrong. They collected 3,843 trajectories from seven frontier models across three scaffolds (OpenHands, MiniSWE, Terminus2) on Terminal-Bench, filtered to 1,794 complete runs, and hand-annotated more than 63,000 execution steps to produce 14 findings.
The headline is the uncomfortable one: failures are overwhelmingly “epistemic” — the agent forms a wrong belief about the environment or the task — and they typically onset within the first few steps but stay hidden until recovery is no longer possible. The agent commits to a wrong mental model early, spends the rest of the run digging, and by the time the failure surfaces in a test result the damage is already structural.
Why it matters: This is the mechanistic version of the “resolved-but-49-messy-steps” problem. If the fatal error is set in the first handful of steps, then end-of-run gates — run the tests, check the diff — fire after the point of no return. The lever is an early belief-check: make the agent state its diagnosis and verify it against the environment before it edits anything, and put the circuit-breaker on the exploration phase, not the final one.
Do These Violent Delights Have Violent Ends? The Post-Merge Fate of Agentic Code#
Xia and Miller (arXiv:2607.09902) tracked agent-authored versus human-authored contributions across 182 repositories and asked the question almost no benchmark asks: what happens after the merge? Merge acceptance rates were comparable between agents and humans — the number everyone quotes. But agent contributions required significantly higher rates of corrective maintenance downstream, and introduced more security and dependency vulnerabilities.
The sharpest number is the interaction with review: each 10-percentage-point increase in a project's no-review rate was associated with roughly 6% higher maintenance burden for the agent code. The cost of skipping review on agent PRs isn't paid at merge time — it's deferred, and it compounds.
Why it matters: Merge rate is a satisfaction metric, not a quality metric — I've said this about the PR-acceptance studies, and this is the longitudinal receipt. If you run an agentic-PR workflow, the metric that matters is corrective-maintenance rate and vulnerability introduction over the following weeks, not the green merge. And that no-review coefficient is a direct rebuttal of the “it merged clean, ship it” habit the reviewer-habituation studies already flagged: the review you skip is a loan against next quarter.
When Does Restricting a Coding Agent to execute_code Help?#
There's a persistent field argument about tool surfaces: give the agent a rich toolbox of file-edit and shell tools, or restrict it to a single execute_code call and let it do everything in the interpreter. Yang, Yu and Desell (arXiv:2607.10569) ran the clean three-arm ablation — baseline (all tools), bash-only, code-only — holding model, harness, and prompts constant, across both Claude Code and OpenAI Codex CLI, on synthetic tasks and SWE-bench Mini.
The result is boring in the right way: pass rates were statistically invariant across tool surfaces. Restricting to code-only was cheaper than — or tied with — its cheapest tool-rich rival in three of four (regime × agent) cells; the one exception, SWE-bench with Claude, showed a directional 14.4% cost increase that wasn't significant. And crucially, the cost differences showed up in cache-adjusted metrics, not in naive per-edit cost.
Why it matters: This lands right next to the “reasoning effort, not tool access” result from earlier this month: the tool surface is a cost-and-latency knob, not a capability knob, at current model sizes. Two consequences — stop expecting a richer toolbox to raise your pass rate, and measure tool decisions in cache-adjusted cost, because that's where the real difference hides. Naive per-edit accounting will tell you the surfaces are equivalent when they aren't.
How Do Practitioners Build SE Agents?#
Most of what we know about SE agents comes from mining repos or studying deployment. Lyu et al. (arXiv:2607.10856) instead went and asked the builders: 20 semi-structured interviews across 12 organizations plus an 80-person survey, focused on how teams actually construct these systems. It's the first study of the construction process rather than the artifact.
The central finding is that when implementation gets cheap, the bottleneck doesn't vanish — it moves. Requirements, coordination, review, and deployment become more visible, and reviewing/evaluating agent output becomes the new central activity. They characterize a seven-stage workflow and a shift they call evaluation-driven development, where evaluation steers iteration and specifications become versioned artifacts read by both humans and agents. They also name six recurring challenges, including unreliable evaluation signals, “comprehension debt” as code outpaces the team's understanding of it, and behavioral drift from provider-side model updates.
Why it matters: This is the qualitative counterpart to everything else in the digest. Every other paper here is a measurement instrument for a stage this study says teams are struggling to evaluate — trajectory quality, post-merge fate, cost surface, oracle strength. “Evaluation-driven development” and “specs as versioned artifacts” match what I keep landing on independently: the spec and the eval harness are the durable assets, the generated code is the disposable output. And “comprehension debt” is the honest name for the thing that makes the post-merge maintenance numbers above so dangerous.
BackendForge: Benchmarking Agentic End-to-End Code Generation with Backend Services#
Guo et al. (arXiv:2607.11042) built a benchmark that refuses to grade snippets: 56 contract-defined tasks derived from real open-source apps, where the agent has to produce a Dockerized backend service that gets tested through live HTTP interactions against an OpenAPI contract. The clever bit is a co-evolution loop — test and code agents iteratively refine the test oracle and the reference implementation together, so the oracle gets progressively stronger instead of staying fixed and weak.
That oracle strength is the whole story. GPT-5.5 hit 55.4% on the base oracle but fell to 28.6% under the final, strengthened oracle — nearly halved. The gap isn't the model getting worse; it's the weak oracle overstating competence, exactly the reward-hacking-adjacent effect that shows up whenever an agent can satisfy a thin test suite without satisfying the actual contract.
Why it matters: This is the backend-services echo of the held-out-tests story from SpecBench and the constraint-decay work: agents can implement individual API behaviors and still fail to assemble a complete, deployable service. If you're building agents that ship backends, your eval's headline number is a function of your oracle's strength, and a co-evolving oracle is a cheap way to stop fooling yourself. Report the number under the strongest oracle you can build, not the one that makes the demo look good.
The Common Thread#
The pass/fail moment is the least informative frame. Every paper here moves the measurement somewhere else — into the trajectory (where the fatal belief forms early), into the weeks after merge (where the maintenance cost lands), into cache-adjusted cost (where tool decisions actually differ), and into oracle strength (where “solved” becomes “half-solved”). A green checkmark is the precondition for measuring quality, not the measurement.
Evaluation is the bottleneck — and it's a design surface. The practitioner study names it outright (evaluation-driven development), and the other four hand you concrete instruments: early belief-checks, post-merge corrective-maintenance tracking, cache-adjusted cost, co-evolving oracles. The teams that win aren't the ones with the best model — they're the ones with the sharpest eval.
The cost of skipping the check is deferred, never waived. Skip the early diagnosis and the trajectory paper says the failure is already baked in. Skip review and the post-merge paper prices it at ~6% more maintenance per 10 points of no-review. Skip a strong oracle and BackendForge shows your 55% was really 29%. The bill always arrives — the only question is whether you see it at merge time or a quarter later.