Scrolling this morning's arxiv drop, I skimmed past the usual pile — a Russian repo benchmark, a web-app-generation benchmark, yet another code-review loop — and stopped on a title that read like a subtweet of the entire field: What Resolve Rate Hides. I've written up a stack of papers this year arguing that SWE-bench pass rate is a lousy headline metric. What makes this one worth a spotlight is that it doesn't stop at the argument — it ships the tool that reads what the pass/fail label throws away. My thesis: a coding agent's trajectory is a first-class diagnostic artifact, and if you build or evaluate agents, you should be mining it, not just scoring the final patch.
What it does#
What Resolve Rate Hides: Trajectory Structure Diagnostics for Coding Agents (Shu et al., July 7) introduces TraceProbe, a framework that takes the raw run of a coding agent — every search, read, edit, command, and reversion — and normalizes it into a canonical nine-type action taxonomy (FILE READ, FILE WRITE, SEARCH, COMMAND, PLAN, NAVIGATE, FETCH, AGENT SPAWN, REASON), each tagged with a deterministic effect label so heterogeneous traces from different harnesses become comparable. On top of that normalized stream it runs two rule-based modules. Insight names single-run anti-patterns lifted straight from ordinary debugging practice: search loops (ten-plus consecutive search/read actions with no write), re-read churn (the same file read three-plus times inside a ten-action window), tool oscillation, and verification skips (no validation command in the final five actions after a write). Converge aligns two runs of the same task and classifies where their behavior diverges under a controlled reference.
The reason this is more than another position paper: I've covered several this year — the “coding benchmarks are misaligned” piece, RigorBench, StaminaBench, SWE-Together — that make the case for process-level evaluation. They argue. TraceProbe operationalizes. The unlock is boring and important: deterministic effect labels plus resolving each benchmark patch hunk down to its enclosing function, so you can ask “did the agent even touch the right function” rather than the much coarser “did it open the right file.” That single change in granularity is what makes the diagnostics bite.
The key result#
The finding I keep coming back to: outcome is not process. Applied to 2,500 trajectories across five production settings on SWE-Bench Verified — Claude Code, Codex, and OpenCode running over Opus 4.6, GPT-5.4, and GLM-5 — the paper shows that even resolved runs differ wildly in how fast they reach relevant code and how much failed work they rack up on the way. File choice barely separates winners from losers; function selection and completion behavior do. Anti-patterns show up more in failures than successes (search loops in ~56% of failed runs versus ~41% of resolved ones), but the honest framing is that they're corpus-level difficulty signals, not per-run death sentences. The number that sticks: in one head-to-head, one agent closed a task in 10 steps with zero failed actions while another ground through 49 steps with repeated failures on the identical issue — two identical green checkmarks, two completely different runs.
Why it matters#
If you build agents, the anti-pattern detectors are cheap, rule-based, and map straight onto instrumentation you can add this week. A search-loop counter — search/read actions since the last write — is the single best signal that your explorer sub-agent is thrashing, and in a sub-agent architecture that's exactly the phase you want a circuit breaker on. A verification-skip flag — did the run end without running a test after its last edit? — is a guardrail that ties directly to the abstain/verify-in-loop thread I keep landing on: it catches the run that “finished” without ever checking its own work. None of this needs a model in the loop. It's counting actions, and the thresholds are in the paper.
If you evaluate agents, Converge is the piece I'd steal. When you're A/B-ing two harnesses or two prompt variants, resolve rate hands you a scalar and hides the mechanism; diffing trajectories on the same task tells you where the better one pulls ahead — reaches the target function twenty steps earlier, skips the re-read churn, doesn't oscillate between read and failed write. That turns “harness B is +3 points” into an actual causal story you can act on. It also reframes ranking: an agent that passes with a third of its actions being wasted work is not equivalent to one that passes lean, even though the leaderboard calls it a tie. Cost, latency, and token burn all live in that gap, and resolve rate is structurally blind to every bit of it.
The caveats#
SWE-Bench Verified again — leakage and saturation make the corpus a shaky substrate, and function-level localization only works because there's a golden patch to resolve hunks against. You don't have that oracle in the wild.
The anti-patterns are difficulty clues, not run-level predictors — the paper is refreshingly clear that a search loop flags a hard task, not a doomed run. Don't wire it as a hard kill switch.
Thresholds (≥10 search/reads, 3+ re-reads, final-5 for verification) are hand-tuned to these five harnesses. The normalization layer is where the assumptions hide, and your trace format may not map cleanly onto the nine action types.
Five settings, all frontier-ish — no small or local models — and rule-based detectors won't catch semantic thrash that doesn't surface as an action-count pattern.
The takeaway#
What I'm filing away: stop treating the agent log as exhaust. The trajectory is the evidence, and a resolved task with 49 messy steps is a bug report, not a win. Concretely, what I'm doing differently this week — adding a search-loop and a verification-skip counter to my own agent traces, and the next time I compare two harnesses I'm diffing trajectories, not just tallies.