← all writing
03 · 16 Nov 2026 · 6 MIN READ

Setup fixes that never reached the script dropped success from 90.7% to 25.9%

90.7% versus 25.9%. That is the gap between two versions of the same environment-setup agent on SetupBench. The only difference is whether a fix the agent found by poking around in a live container gets written back into the setup script before it moves on.

Every agent builder has watched this happen. The agent hits a missing system library, runs apt-get install interactively, the tests go green, and the session declares victory. Then you rebuild the container from the Dockerfile the agent produced, and it falls over on the same library. The fix lived in the container, not in the artifact.

The paper that put a number on this is HerAgent: A Block-Level Checkpointing Harness for Coding-Agent Environment Setup by Xiang Li, Siyu Lu, Federica Sarro, Claire Le Goues and He Ye. Its headline is about checkpointing granularity. I think its most useful result for practitioners is buried in the ablation table.

Commands are too small to undo, scripts too big to repair

Existing setup agents manage progress at one of two levels. Command-level harnesses like ExecutionAgent and Repo2Run advance one shell command at a time. The authors say this design "can struggle to recover from earlier decisions". Whole-script harnesses like PIPer and Installamatic treat the entire Dockerfile or Bash script as the unit, which gives clean trajectories but makes failures hard to localise.

Their motivating example is very familiar. In DataHub, Node 17 gets installed at command #7. It ships with OpenSSL 3, which breaks the frontend's WebPack build, but nobody notices until frontend validation at command #14. A command-level agent then spends roughly twelve more steps (#17 to #28) cleaning up. A whole-script agent starts over from scratch.

HerAgent's unit sits in between: a semantic setup block, meaning a group of dependent commands that implements one functional stage (system packages, language runtime, project dependencies, native build config, test tooling, and so on) and ends at a validation check. The loop goes like this:

  1. Inspect the repository read-only (ls, find, grep, cat) and have the LLM emit a block sequence as JSON.

  2. Run blocks in order. A block is checkpointed via docker commit only when both its commands and its validation pass.

  3. On failure, restore the previous checkpoint, let the agent experiment in the live container, then patch the fix back into the block and replay the block from the clean checkpoint. Only a passing replay counts.

  4. If the evidence points upstream, roll back to an earlier block and invalidate everything after it.

  5. At the end, replay the whole block sequence in a fresh container using only the artifact.

Step 5 is the honest part. Any setup that only worked because of hidden live-repair state gets rejected.

Pull the patch-back step and success collapses to 25.9%

The ablations on SetupBench with gpt-5.4 split success into three checkpoints: forward progress, block replay and final clean replay. Read them as a funnel:

Without patch-back, the agent believes it is making progress on two-thirds of tasks, yet barely a quarter of those environments survive being rebuilt. It is also the cheapest variant by a wide margin, which is exactly why it is dangerous: it looks efficient right up until somebody runs the artifact.

The rest of the table backs up the paper's main thesis. Arbitrary 8-way chunking loses 42.6 points against semantic boundaries, so it is not just "checkpoint more often". Where you cut matters.

The cost numbers flatter it more than the success numbers

Across four datasets and six baselines, HerAgent wins every comparison. The cleanest one uses the same model on both sides: on SetupBench, OpenHands with gpt-4.1 reaches 46.3% at $2.20 per task, while HerAgent with gpt-4.1 reaches 70.4% at $0.12. That is the 94.5% cost reduction in the abstract, and it comes with a 24-point accuracy gain. On Installamatic-Bench with gpt-4o-mini it goes from 52.0% to 85.0% at two cents a task.

Not every comparison is that tidy, and I would read the table with a few things in mind:

The scope is also narrower than "environment setup" sounds. The authors exclude tasks needing private credentials, paid services, GPUs or distributed deployment, and they note that a passing readiness command does not prove the repo is fully usable. That rules out a lot of the setup pain I deal with in client work.

Where the remaining failures live

They hand-labelled 904 failures (κ = 0.601 before adjudication). Incomplete dependencies account for 45.6% and dependency conflicts for 17.5%, so dependency problems make up 63.1% in total. Project-dependency and test-tooling blocks absorb 64.9% to 93.5% of all local repairs. The authors' summary of what is left is fair:

remaining failures "point beyond progress control"

Better checkpointing will not fix a resolver that cannot reason about compatibility across the whole dependency graph.

A setup fix that isn't in the artifact doesn't count

This applies well beyond environment setup. Any agent loop that experiments in a mutable workspace (Claude Code in a devcontainer, a sub-agent fixing CI, a migration agent poking at a staging database) has the same failure mode: state that changed in the world but never made it into the artifact you will actually ship.

Here is what I am taking from it:

My bet is that "final clean replay" becomes a standard eval column for setup and infra agents within a year. Without it, a benchmark measures how well the agent can talk a container into working, not whether anyone else can rebuild what it produced.


Working on something similar?

Say hello — I read every email.