86.1% of tasks completed. Backup taken before the destructive step in 15.2%. Those two numbers come from the same agents on the same runs, and they are the whole argument of Back Up Before You Regret, a new paper from Northeastern, Berkeley, Chicago and UIUC. I have been treating "it passed the task" as a proxy for "it is safe to let it near my data". This paper says that proxy is broken.
Capable and careless turn out to be independent axes#
The authors built RegretBench: 205 executable multi-turn tasks derived from real regret reports scraped from X, GitHub and Stack Overflow, spanning terminal, SQL, browser, email and DevOps tools. Each task contains a "regret moment", an action after which the user would wish they had a copy of what was there before. The metric, RMBR, is the share of tasks where the agent backed up before that moment.
They ran six models across their native harnesses: Claude Opus-5 and Sonnet-5 in Claude Code, two GPT 5.6 variants in Codex, plus Gemini 3.7-Flash and DeepSeek V4-Pro in Hermes. The spread is wide:
Average across agents: 86.1% completion, 15.2% backup rate.
GPT 5.6-Sol: 5.9% backup rate. DeepSeek V4-Pro: 2.9%.
Opus-5 was the best of the group and still missed the regret moment in 35.6% of tasks.
The part that stuck with me is the generational comparison. Going from GPT 5.4 to 5.6-Sol, their capability index rose from 39 to 47 while RMBR stayed flat at 5.0 to 6.7%. The authors put it bluntly:
task completion ability does not imply backup awareness
So waiting for the next model to get more careful is not a plan. Nothing in this data suggests carefulness arrives as a side effect of capability.
Why does this happen? The paper cannot say, and I will not pretend to. My own guess is that backup is never part of the request. Nobody writes "and keep a copy" in a task, so a model optimizing for the stated goal has no reason to spend a tool call on it. The authors point at the same difficulty: it is hard to predict which changes will be regretted, so an agent that backs up everything pays a constant tax and one that backs up nothing pays rarely but badly.
Swapping the harness barely moved the number#
My first instinct was that this is a prompting problem and a better system prompt would fix it. A 60-task ablation says otherwise: Opus-5 stayed in a 68.3 to 73.3% band across harnesses, Sol in 5.0 to 8.3%, DeepSeek in 0 to 1.7%. The model dominates. That matches what I see in practice, where an instruction like "be careful with destructive commands" gets read once and then competes with fifty tool results for attention.
It also means Claude Code users on Opus-5 are on the favorable end of this chart, and that is exactly where I would not relax: a third of tasks still missed.
A rule table beats asking the model to remember#
The fix, BackUpper, is deliberately unglamorous. It sits between the agent and its tools and intercepts every call before it executes, in four tiers:
Rewrite the command into a safer form (for example, move instead of delete).
Pair the command with a backup step that runs first.
Snapshot the workspace or native-tool state when a rewrite is not possible.
For everything ambiguous, call a lightweight LLM checker and a backup subagent.
The clever bit is that subagent decisions get distilled into new rules, so the expensive tier shrinks over time. Rule coverage went from 14.9% to 31.2% after self-reflection on 30 Terminal-Bench 4 tasks. This is the same pattern I like in permission hooks: let the model be the slow path, and compile what it learns into the fast path.
Users can also narrow the scope with preferences (High-Impact, Beyond-Request, Human-Work modes), and those reached 84.5% average F1 against human labels about what deserves a backup. That is the knob I would want, because "back up everything" is exactly how you end up with the 8.9% of tool calls Opus-5 spends on self-backup when left to its own devices.
The numbers on cost are the reason I would actually try it:
Covered every regret-moment action in the benchmark.
Restored state reached 92.2% character-level similarity to the original, versus 14.3% with no restoration and 33.8% with post-hoc recovery.
$0.014 per task, against $0.344 for a skill-based baseline that asks the agent to back itself up. Tokens fell from 382K to 55K per task.
On 40 external tasks, out-of-workspace backup rates rose by 76.9 points for Sol and 52.9 for Opus-5, with pass rates unchanged.
That last line matters most. Safety layers usually tax success. Here the pass rate did not move.
One more detail: four subagents averaged 94.6% RMBR in their ablation, so the expensive tier is not a weak link when the model behind it is strong. That is a reminder that this design transfers the judgment from a distracted main agent to a focused one with a single job.
What I do not buy yet#
Regret moments come from human reports, so they capture what people regret, not necessarily how agents fail differently. The evaluation is black-box; the authors cannot say why models skip backups. The subagent tier leans on subagent quality and may weaken with smaller models. And some state, like third-party replies or server-generated fields, "may not be restorable even from a complete backup", which is the honest limit of any snapshot approach. An email already sent stays sent.
Cost realism is the other question. $0.014 per task is measured on a benchmark of short, tool-heavy tasks. A long-running refactor that touches thousands of files makes the snapshot tier far more expensive than a rewritten rm, and the paper does not show that regime.
There is also a judging caveat: Claude Sonnet-5 acts as both task judge and backup judge. It agreed with human annotators on 95.3% of 700 sampled actions, which is good, but I would still like a second judge family.
What changes in my setup#
I am going to stop counting a green run as evidence of safe behavior and add a second column to my own agent evals: did state-changing steps have a recoverable prior state. And in Claude Code I will move the backup requirement out of CLAUDE.md prose and into a pre-tool-use hook that snapshots before delete, overwrite, migrate and send operations, since the paper shows the model will not reliably volunteer it.
My prediction: within a year, pre-execution interception layers become a standard part of agent harnesses, and "restorable" joins "passes" as a reported eval metric.