Kill the run that looks doomed and start over. It is the cheapest-sounding trick in agent engineering, and I have wired it into harnesses more than once. Srinivasan, Li, Menzies and Assunção tested it on 147,002 recorded SWE-agent and OpenHands traces. Continuing beat restarting in 216 of 224 configurations, on both solve rate and token cost.
A predictor that works on average and fails on the one question you care about#
The traces come from four open-weight setups: OpenHands with Qwen3-Coder-480B (67,074 runs), SWE-agent with Qwen3.5-122B (15,098), OpenHands with Qwen3.5-122B (30,957) and OpenHands with MiniMax-M2.5 (33,873). The decision point is 20k transcript tokens. By then 87.5% of runs have executed tests, but only 32.8% have edited any code.
Pooled across all tasks, a failure predictor looks usable. Counting what the agent did in the prefix lifts AUC from 0.57 to 0.61. That is weak, but it is above chance. The catch is what it measures. Task difficulty is hugely clustered, with an intraclass correlation of 0.71 to 0.82. A model that just knows which tasks are hard scores well without reading the agent's behaviour at all.
So the authors compare attempts on the same task, where one run succeeded and another failed. That is the situation a restart policy faces: this attempt versus a fresh one. There the numbers collapse:
Count model at 5k tokens: within-task AUC 0.50
Task-conditional model at 20k: 0.50, over 38,447 pairs from 1,737 mixed-outcome tasks
A LoRA-tuned Qwen3-0.6B monitor at 20k: 0.50, over 32,406 attempts and three seeds
An oracle that knows each task's solve rate: pooled AUC 0.97, within-task 0.50 by construction
That last line is the whole paper in miniature. A near-perfect pooled score and zero information about which attempt is going to work.
I find the permutation and equivalence testing reassuring. The authors ran unseen-repository cross-validation, permutation controls and TOST equivalence tests, and they flag that the SWE-agent/Qwen3.5 permutation control drifted from 0.50, so they make no formal equivalence claim there. That is the right level of care, and it makes the 0.50 results more believable than a single lucky split would.
It also changes how I read my own dashboards. I have looked at a plot of step count versus eventual success and felt I could see doomed runs. What I was seeing was that hard tasks take more steps. The agent was not telling me anything about itself.
Restart lost even when the signal was handed an easy win#
They replayed the recorded attempts under shared token caps from 30k to 100k, comparing learned restart, matched random restart and plain continuation. At a 75k cap, continuation beat all 56 restart configurations on solve rate and mean cost. For SWE-agent with Qwen3.5, continuing solved 28.71% at 68.10k mean tokens, while learned restart solved 20.66% at 69.93k.
The learned-versus-random gap ranged from −0.66 to +1.98 points. An exact decomposition shows why that gap is not real skill: the between-task term drove the sign in every setting (−0.70 to +1.75), while the within-task term stayed between −0.11 and +0.24. The scorer sometimes restarted easier tasks and got credit for it.
uninterrupted continuation achieves higher solve rates at lower mean token cost than both restart policies
There is a cost realism angle too. A restart throws away every token already spent, and the replay charges for that. Continuation was cheaper on average because a restarted run pays for a fresh exploration phase and still has to clear the same task. With uncapped budgets, learned and random restart differed by under 0.20 points, which means the policy was not buying anything at all.
Where I would still reach for the restart button#
This is a replay on recorded runs, so it says nothing about a restart that preserves state or injects what the first attempt learned. The authors say their monitor omits the intermediate test supervision and state-preserving recovery of the earlier FailFast–RestartSmart work. It is also four open-weight setups, mostly Python, and token counts ignore caching discounts and latency.
Later signals do exist. Within-task AUC creeps to 0.56 by 60k tokens, but only 23% of runs are still going by then, so there is little budget left to save. Reference-hunk recall at completion reaches 0.62. That tells you about a run that is already finished.
What I am changing in my harness#
Concretely, I would do three things this week:
Remove any abort-and-retry rule keyed to early step counts, tool mix or first test failures.
Evaluate any new stopping rule on pairs of attempts from the same task, not on a pooled AUC.
Spend the saved effort on the scaffold and on giving a stuck run better context, rather than on a fresh start.
I will stop treating early-prefix signals as a reason to abort. Step counts, tool-call mixes and early test failures all predict which task you are on, not whether this attempt will recover. If I want a restart rule, it has to be validated within task, with attempt-level discrimination and net compute savings against simply continuing. That is the bar the authors propose.
There is a side result worth a footnote. On shared Qwen3.5 tasks, SWE-agent solved 47.30% against OpenHands' 33.24%, a 14.06 point gap. If you have compute to spend on tuning something, the scaffold looks like a better target than a failure oracle. I expect the next serious attempt at early stopping to look at state-preserving recovery, not at scoring the first 20k tokens.