A step that follows a passed step passes 70.2% of the time. A step that follows a failed one passes 14.1%. That single pair of numbers from Multi-step SWE tells you most of what you need to know about running a coding agent on a chain of dependent tasks: the chain does not forgive, and the agent does not seem to know it is already in trouble.
The benchmark, from Klavis AI, is 69 tasks, each a chain of 5–15 dependent steps (574 steps in total, median 8). Every step is a multi-hour engineering request, and the workspace carries forward, so the agent keeps building on its own code. Families are optimization, feature implementation, version upgrade and migration, mostly in C/C++. A rollout only counts as solved if every step gets full credit.
The best model solved 18.5% of chains#
Nine models ran through Harbor harnesses (Claude Code, Codex CLI, mini-swe-agent, Grok Build), four rollouts each, at maximum reasoning effort. Top of the table:
Claude Opus 5.5: 18.5% solve rate, 31.9% Pass@4, 56.8% of steps passed
Claude Fable 5.1: 17.8% solve rate, 49.1% of steps passed
Gemini 4 Argon and GPT-6.1 Sol: 10.1% each
Grok 4.7 at the bottom: 1.4%
The gap between 56.8% of steps and 18.5% of tasks is compounding. The share of rollouts still fully on track falls from 34.9% after the first step to 9.0% at the end. Opus 5.5 goes from 54.7% to 19.2%. Also, 38 of 69 tasks were solved by no model at all, including 16 of the 20 version-upgrade tasks. Pooled family scores are 15.0% for optimization, 13.9% for migration, 5.1% for features and 2.9% for upgrades.
Note the bootstrap interval on Opus 5.5's solve rate: [11.6, 25.7]. With 69 tasks, the ranking of the top two is not something I would bet on.
The authors also spent effort on verifier integrity, which I appreciate. They attacked their own early verifiers (a reward file rewritten by a background process, cached outputs returned for repeated inputs, a planted test runner) and added a check for each, then graded on hidden inputs with unprivileged execution. Two rounds of expert review, by engineers with at least five years of experience, came on top. When a benchmark reports that agents are bad at something, I want to know the grader cannot be gamed, and this one at least tried.
Nobody ran out of time#
The part I keep returning to is why steps failed. Only 12% of failed steps ran to within five minutes of their budget. For six of the models, the median failed step used just 24–37% of the time it was given. And passing and failing sessions for the same step and model have a median duration ratio of 1.05, so the clock is not what separates them. Agents stop when they decide to stop.
Then the final reports. In 52% of failed steps the agent said it was done and named no shortcoming. In another 26% it listed specific failures and stopped anyway, with a median of 66 minutes left. Only 3% blamed the budget.
A completion report is almost no evidence#
You might hope to use the agent's own report as a gate. The paper says no: agents report completion with no named shortcoming in 73% of passed steps, but when they do that, the verifier still fails 57% of the time, against 65% across all steps. That is a small shift for a statement that sounds like a verdict.
In several failures the session itself contained the defect and the agent dismissed it. Two examples from the paper: NaN output where the documentation specifies 0, and an IMAP MOVE argument order that contradicted RFC 6851.
The tests the agent trusted were the wrong ones#
Agents often stopped once their own tests or the public tests passed. Passing runs were more likely to use an independent oracle or a differential comparison against a reference. One CSS task makes the point: the public tests included 2 float-layout tests, the hidden ones 22.
It also matches the optimization numbers. 94.3% of 4,068 optimization step records passed the correctness gate, yet 88.2% of failed optimization steps produced correct output and simply did not beat the reference speedup. The agent got to correct and stopped, without a target to push against.
What I am changing in my own harness#
First, I will stop treating "agent says done" as a stopping condition and make it a trigger for an independent check: a differential run against a known implementation, or tests the agent did not write. Second, if a step fails, I will not let the next one build on it silently. At 14.1% pass rate after a failure, that is mostly wasted compute. Third, I will give agents an explicit numeric target and the remaining budget, since early stopping rather than exhaustion is the dominant failure.
There is a design lesson underneath this. A loop that ends when the model emits a final message is a loop whose quality depends on the model's self-assessment, and this paper puts a number on how weak that self-assessment is. In a Claude Code style setup I would put the verifier outside the agent's context: a hook or sub-agent that runs the independent check and feeds the result back, so the main agent cannot talk itself out of a red result.
Caveats are real. Language coverage is uneven (38 of 69 tasks are C/C++), memorization from public projects cannot be ruled out, the optimization threshold is the reference solution's speedup, and the authors list an isolated-step baseline as future work, so we cannot yet separate compounding errors from step difficulty. Still, my prediction: the next useful agent benchmarks will report how often an agent stops early with budget left, not just whether it passed.