90.7% versus 25.9%. That is the gap between two versions of the same environment-setup agent on SetupBench. The only difference is whether a fix the agent found by poking around in a live container gets written back into the setup script before it moves on.
Every agent builder has watched this happen. The agent hits a missing system library, runs apt-get install interactively, the tests go green, and the session declares victory. Then you rebuild the container from the Dockerfile the agent produced, and it falls over on the same library. The fix lived in the container, not in the artifact.
The paper that put a number on this is HerAgent: A Block-Level Checkpointing Harness for Coding-Agent Environment Setup by Xiang Li, Siyu Lu, Federica Sarro, Claire Le Goues and He Ye. Its headline is about checkpointing granularity. I think its most useful result for practitioners is buried in the ablation table.
Commands are too small to undo, scripts too big to repair#
Existing setup agents manage progress at one of two levels. Command-level harnesses like ExecutionAgent and Repo2Run advance one shell command at a time. The authors say this design "can struggle to recover from earlier decisions". Whole-script harnesses like PIPer and Installamatic treat the entire Dockerfile or Bash script as the unit, which gives clean trajectories but makes failures hard to localise.
Their motivating example is very familiar. In DataHub, Node 17 gets installed at command #7. It ships with OpenSSL 3, which breaks the frontend's WebPack build, but nobody notices until frontend validation at command #14. A command-level agent then spends roughly twelve more steps (#17 to #28) cleaning up. A whole-script agent starts over from scratch.
HerAgent's unit sits in between: a semantic setup block, meaning a group of dependent commands that implements one functional stage (system packages, language runtime, project dependencies, native build config, test tooling, and so on) and ends at a validation check. The loop goes like this:
Inspect the repository read-only (ls, find, grep, cat) and have the LLM emit a block sequence as JSON.
Run blocks in order. A block is checkpointed via docker commit only when both its commands and its validation pass.
On failure, restore the previous checkpoint, let the agent experiment in the live container, then patch the fix back into the block and replay the block from the clean checkpoint. Only a passing replay counts.
If the evidence points upstream, roll back to an earlier block and invalidate everything after it.
At the end, replay the whole block sequence in a fresh container using only the artifact.
Step 5 is the honest part. Any setup that only worked because of hidden live-repair state gets rejected.
Pull the patch-back step and success collapses to 25.9%#
The ablations on SetupBench with gpt-5.4 split success into three checkpoints: forward progress, block replay and final clean replay. Read them as a funnel:
Full HerAgent: 96.3 → 96.3 → 90.7, at $0.308 per task.
Without patch-back: 66.7 → 37.0 → 25.9, at $0.074.
Without block partitioning (one whole script): 55.6 → 53.7 → 46.3, at $0.372.
Fixed 8 chunks instead of semantic boundaries: 72.2 → 68.5 → 48.1.
Command-level rollback instead of block rollback: 70.4 → 68.5 → 53.7.
Regenerating the failed block instead of repairing it locally: 96.3 → 90.7 → 74.1.
Without patch-back, the agent believes it is making progress on two-thirds of tasks, yet barely a quarter of those environments survive being rebuilt. It is also the cheapest variant by a wide margin, which is exactly why it is dangerous: it looks efficient right up until somebody runs the artifact.
The rest of the table backs up the paper's main thesis. Arbitrary 8-way chunking loses 42.6 points against semantic boundaries, so it is not just "checkpoint more often". Where you cut matters.
The cost numbers flatter it more than the success numbers#
Across four datasets and six baselines, HerAgent wins every comparison. The cleanest one uses the same model on both sides: on SetupBench, OpenHands with gpt-4.1 reaches 46.3% at $2.20 per task, while HerAgent with gpt-4.1 reaches 70.4% at $0.12. That is the 94.5% cost reduction in the abstract, and it comes with a 24-point accuracy gain. On Installamatic-Bench with gpt-4o-mini it goes from 52.0% to 85.0% at two cents a task.
Not every comparison is that tidy, and I would read the table with a few things in mind:
Aggregation varies. HerAgent reports any@3. Installamatic's baseline is any@10, which actually favours the baseline, so the 33-point gap is if anything conservative.
Time columns are not comparable. ExecutionAgent's 74 minutes was measured with five projects in parallel, and Repo2Run's time and steps cover only its 361 successful builds.
The authors say some benchmarks lack full prompts, commits or validation commands, so part of the gap may be version drift. Package registries also shift under everyone.
On Repo2Run-Bench with gpt-4o, the win is 87.1% vs 86.0%. The headline there is cost ($0.42 vs $3.56), not accuracy.
The scope is also narrower than "environment setup" sounds. The authors exclude tasks needing private credentials, paid services, GPUs or distributed deployment, and they note that a passing readiness command does not prove the repo is fully usable. That rules out a lot of the setup pain I deal with in client work.
Where the remaining failures live#
They hand-labelled 904 failures (κ = 0.601 before adjudication). Incomplete dependencies account for 45.6% and dependency conflicts for 17.5%, so dependency problems make up 63.1% in total. Project-dependency and test-tooling blocks absorb 64.9% to 93.5% of all local repairs. The authors' summary of what is left is fair:
remaining failures "point beyond progress control"
Better checkpointing will not fix a resolver that cannot reason about compatibility across the whole dependency graph.
A setup fix that isn't in the artifact doesn't count#
This applies well beyond environment setup. Any agent loop that experiments in a mutable workspace (Claude Code in a devcontainer, a sub-agent fixing CI, a migration agent poking at a staging database) has the same failure mode: state that changed in the world but never made it into the artifact you will actually ship.
Here is what I am taking from it:
Every setup or infra fix an agent makes has to be written back into the Dockerfile, devcontainer config or script before the next step, and the gate is a replay from a clean snapshot, not the live container's exit code.
Cut checkpoints at functional boundaries with their own validation (runtime installed, deps resolved, tests discoverable), not after every command and not only at the end. Container snapshots make this cheap.
When something breaks late, ask which earlier stage owns the failure before letting the agent pile fixes on top. The Node 17 example is the pattern to watch for.
My bet is that "final clean replay" becomes a standard eval column for setup and infra agents within a year. Without it, a benchmark measures how well the agent can talk a container into working, not whether anyone else can rebuild what it produced.