The scariest number in this paper is not the 53.5-point drop. It is that in 58.9–88.2% of failed runs, the agent finished with exactly the same tests failing as when it started. It looked at someone else's broken work and signed off.
That is the setup of SWE-Residual, from Jinze Li and colleagues. Most coding benchmarks hand an agent a clean repo and a request. Real workflows rarely look like that: a previous agent run, a teammate or a crashed session leaves a state marked "done", and the next agent picks it up. The authors call the ability to cope with that Starting-State Robustness, and they measure it directly.
Same request, same tests, only the starting state changes#
They collect 198 Residual States from 111 tasks in DeepSWE and Terminal-Bench. Each one is what a predecessor agent (DeepSeek-V4-Flash, Qwen-3.8-27B or GLM-5.3) left behind after declaring the task complete while the hidden tests still failed. Two human reviewers filtered the states. The agent under test sees the original request and the state, but no trajectory and no hint that anything failed. Because request and verifier are unchanged, the starting state is the only variable.
Five frontier agents were evaluated. On DeepSWE the drops from Clean Start to Residual Start range from 20.6 to 53.5 points; on Terminal-Bench from 20.6 to 34.9. A few rows:
Gemini-3.8-Flash on DeepSWE: 72.2% clean, 20.0% residual
GLM-5.3-Flash on DeepSWE: 68.9% clean, 15.4% residual
GPT-6-Sol on Terminal-Bench: 71.4% clean, 36.5% residual
The states are not garbage either. The median DeepSWE Residual State already passes 98.4% of tests. The agent is one small step from done and still fails most of the time.
Acceptance bias: the inherited work becomes the spec#
The authors name the failure mode acceptance bias:
agents treat the inherited work as the definition of what remains to be done instead of checking it against the request.
The evidence is consistent. Failed continuations keep 86.4–98.6% of the lines the predecessor wrote, take 0.56–0.84 times as many steps as clean runs (GPT-6-Sol is the exception at 1.02), and repair only 10.4–20.1% of the initially failing tests. The same agents pass 72.8–88.4% of those tests from a clean start, so the capability is there. It just does not get used.
One case study says it all: Gemini-3.8-Flash passes all 11 failing tests of the onedump task from scratch. From the residual state it submits after 43 steps without changing a single file, describing the inherited work as its own.
There is also a pattern with files. Agents opened the files the reference solution needed in 37.6–64.0% of residual runs and still did not edit them. They looked at the problem and did not see it.
Leaderboards flip when you start from a mess#
On DeepSWE, Gemini and GLM rank first and second from a clean start. With a residual start they drop to third and fourth. GLM leads Qwen by 17.8 points clean; Qwen leads GLM by 15.2 points residual. If your product is an agent that resumes work, a clean-start leaderboard is picking the wrong model for you.
The one agent that held up better, GPT-6-Sol, did something specific: in 11 of 30 sampled failed continuations it named the defect and tried to repair it, versus at most 2 of 30 for the other three. That is a behavior, not a size effect, which suggests it is trainable or at least promptable.
What I would change in a handoff loop#
The paper reports no mitigation experiments, so what follows is my inference, not a result. But the failure is so specific that the fixes are obvious candidates:
Do not give the next agent only the state. Give it the original request plus an instruction to re-derive the acceptance checks from the request before reading the diff.
In a spec-driven flow, make the spec the artifact the verifier agent reads first, and the previous agent's summary the last thing it reads, if at all.
Treat "previous run said done" as a claim. In sub-agent setups where a reviewer picks up a worker's output, have it run the checks itself and require a failing-test list before it may approve.
Measure it. Add a residual-start slice to your own evals by saving the almost-right states your agents produce.
I also like that the benchmark isolates one variable. Most agent evals bundle context, tools, tests and starting state together, so a score change is hard to attribute. Here the request and verifier never move, which makes the 20-point floor attributable to the inherited state alone, and that is what makes the result useful for deciding where to spend effort in a harness.
A cheap check falls out of the numbers: if an agent ends with the identical failing test set it began with, that is a strong signal it never engaged. That is detectable without any judge model.
How far I would trust it#
The scope is narrow: 198 states, two source benchmarks, three predecessor families, all agent-written. Human-left work and other handoff styles are not covered. The acceptance-bias analysis runs only on DeepSWE because it needs per-test outcomes and final patches. The agents tested are mostly the cheaper "Flash" tier, and shell history left in a Terminal-Bench state may leak hints about the predecessor. A failing test can also hide a lot of work, since suite granularity differs.
Even so, the direction is hard to dismiss. A 20-point floor on the gap across every agent and every predecessor group is not noise. My bet is that resumption quality becomes a standard eval axis within a year, because every long-running agent product depends on it.