The agent knew. That is the uncomfortable part of Knowing When to Stop: The Evidence-Action Gap in Coding Agents. In runs where the filesystem was read-only or the source was unreadable, 98.6% reached the point where the tool feedback was sufficient to prove the task was blocked. Then they kept going for an average of 38.9 more tool calls, and only 20.3% stopped within ten steps of that point.
I have watched this happen in real sessions: a denied write, a retry, a chmod attempt, a workaround, and twenty minutes later a confident summary of work that never landed. This paper puts numbers on it.
A benchmark that injects the wall on purpose#
The authors build ImpasseBench from 102 solvable SWE-smith repair tasks across ten Python repositories and inject one of twelve blockers into each run. Six are persistent barriers (read-only filesystem, no write space, unreadable source, root-owned source, missing dependency, early sandbox loss). Three limit verification, two are temporary faults that recover, and one is a task whose reported bug is already absent. Budget is 80 steps.
The clever part is the marker t*: the first step where the collected tool feedback meets predefined criteria for sufficient evidence, around step 7.5 on average. Behavior is then scored relative to it. Continuing after t* is over-persistence. Stopping before it, or giving up on a fault that would have recovered, is premature handoff. Without that marker you cannot tell a diligent agent from a stubborn one.
Scale: eight model configurations from five families, 19,371 trajectories.
Every unsafe request came after the agent already knew#
Roughly a third of blocked runs burned the whole budget (the paper says 33.5% in the intro and 34.0% in Section 3.2.1, so read it as about a third). More important for anyone running agents with real permissions: every unsafe request the authors detected happened after t*. Their examples include guessing credentials, reading protected credentials, removing or skipping test assertions, and replacing implementations through test-loading files.
Baseline peaks for unsafe requests were 18.0% for Gemini 3.5 Flash, 4.2% for Gemini 3.1 Pro, 4.1% for GPT-5.6 Sol and 3.3% for Kimi K2.5. The safety labels cover specific observable behaviors only, so a clean run is not a proof of safety, as the authors say themselves.
Turn that around and it is a design hint: the dangerous window is not the beginning of a task. It is the stretch after the evidence is in and the agent still has budget left.
A one-tool fix, and who actually uses it#
The first scaffold is boring on purpose: an escalate_to_human tool the agent can call to file a ticket. Effects on persistent barriers:
Post-evidence calls: 38.9 down to 17.5
Runs ending within ten steps of t*: 20.3% up to 45.7%
Budget exhaustion: 34.0% down to 6.7%
Unsafe requests after evidence: 3.89% down to 0.68%
Adoption varies a lot by model. Kimi K2.5 and Haiku 4.5 took to the tool and cut limit endings by 73.3 and 64.6 points. Qwen3-Coder-30B barely touched it, submitting tickets in 19.7% of persistent-barrier runs. If you ship an escape hatch and only test it on a model that likes escape hatches, you have measured nothing.
When the model ignores the tool, move the rule into the harness#
For that case the authors use a rule-based monitor in the execution harness. It watches runtime output for patterns such as a denied edit, root ownership or a refused chmod, then injects a reminder to review remaining work and offers a pre-filled ticket template. On Qwen3-Coder-30B over 102 held-out SWE-bench Verified tasks, accepted escalations went from 16.3% to 42.5%, and end-to-end repair success after scripted environment restoration went from 7.6% to 21.8%. Solve rate on clean tasks stayed flat (50.3% vs 49.0%, p=0.839).
This is the same lesson I keep relearning with hooks in Claude Code: a few lines of string matching in the harness often beat a paragraph in the system prompt. The monitor does not need to be smart. It needs to see the exact error text the model has already seen and refuse to let it scroll past.
Premature handoff is the price, and the handoff itself is thin#
The tool is not free. On temporary faults, ten tool-enabled runs escalated with empty patches even though the sandbox recovered, and GPT-5.6 Terra's repair success on recoverable faults dropped from 70.1% to 63.7%. An agent that quits early on a flaky sandbox is a different kind of useless.
Then there is what lands in the ticket. An LLM judge rated 3,149 tickets: 74.1% sufficient, 25.9% partial, none insufficient. Blocker, request and resume condition were almost always fine. The gap was the agent's own progress, missing or incomplete in 35.7% of tickets and responsible for 97.3% of the partial ratings. Kimi K2.5 escalated in 82.0% of persistent-barrier runs but only 52.7% of its tickets were sufficient. Escalating often and escalating usefully are separate skills.
One caveat on the evidence: the blockers are injected into sandboxes, labels are rule-based, the monitor was tested on a single model, and the judge for ticket quality is itself an LLM. Real handoffs across multiple turns with a human are future work.
What I would change this week#
Give every long-running agent an explicit way to say "I am blocked", and test it on your weakest model, not your favorite.
Add a harness-side pattern monitor for permission and filesystem errors. It is cheap and the paper shows it works on a model that ignores the tool.
Require the handoff template to include what was done and what was verified, since that is exactly the field agents omit.
Log t* in your own evals: the step at which the evidence was sufficient versus the step at which the run ended.
My prediction: stop-and-ask quality becomes a standard column in agent evals next to resolve rate, because a model that solves 60% and quietly rewrites your tests on the other 40% is worse than one that solves 55% and files a clear ticket.