Tuning your agent's harness offline made it slower. On 50 held-out performance-optimization tasks, a frozen harness reached a 1.442× speedup in four hours. The same harness after a train-and-validate evolution pass reached 1.281×, at a higher cost per run. The BoxBox paper puts those numbers next to a version where the agent edits its own harness in the middle of a run, and that version got 1.702× for less money than the frozen baseline.
I have spent plenty of evenings tweaking system prompts and tool descriptions between runs. This paper is a fair argument that I was optimizing at the wrong moment.
What BoxBox actually adds to a normal agent loop#
The setup is PerfRace: 100 long-running optimization tasks on real software projects, split 40 train, 10 validation and 50 held-out test. Each test task gets three runs with a four-hour budget that covers execution, harness changes and evaluation together. Everything starts from the same initial harness, Terminus-KIRA, and one model family with high reasoning effort.
The mechanism is small. The agent gets a boxbox() tool. When it calls it, a kernel pauses the run and keeps the trajectory, terminal session and workspace intact. The agent then enters a review session where it can rewrite any part of its executable harness: prompt rules, skills, tools, execution logic, fault handling. The kernel validates the new commits, hot-reloads the runtime, and rolls back if activation fails. Runtime faults can trigger a review too, and every revision is a Git commit, so later reviews can inspect earlier ones.
Cost of all this is modest. The median run had 5 reviews, 92% had between 3 and 6, and reviews used a median of 2.1% of the time budget and 13.8% of model cost.
Offline evolution lost to doing nothing#
Frozen harness: 1.442×, $23.22 per run, 0.2142 speedup per dollar
Harness Evolution (offline): 1.281×, $24.56, 0.1243
Harness Scaling: 1.539×, $12.98, 0.2872
BoxBox: 1.702×, $16.43, 0.4886
Two things stand out. First, offline evolution is below the frozen baseline, and the authors say so bluntly:
Offline harness optimization does not reliably improve performance.
Second, the bill. The paper reports that offline training and validation add at least $33.73 per test run on top of that, and flags this as a lower bound because some usage records are missing. So the approach I would have reached for first is both the least accurate and the most expensive. BoxBox cuts mean model cost about 29% versus frozen while gaining roughly 0.26× of speedup.
The paper's explanation is implied by the design rather than proven: a harness edit made offline is tuned to other tasks, while one made mid-run sees the actual terminal, the actual failing build, the actual profiler output. I find that plausible, but nothing in the numbers isolates it.
The Harness Scaling row deserves a note too. It beat frozen on both speedup and cost, so spending the budget differently, without any offline training, already helped. BoxBox then added roughly 0.16× more speedup on top, at a few dollars more per run. Read together, the pattern is that how you spend the four hours matters more than how much you pre-invest in a better starting harness. For anyone budgeting an agent run, that is the number worth putting in the planning doc.
Both knobs mattered, though the evidence is thin#
The ablations are on 14 sampled test tasks, so treat them as directional. Full BoxBox scored 1.584× there. Replacing agent-triggered review with a review every 30 minutes dropped it to 1.508× at a much higher cost ($21.58 against $15.07). Restricting edits to a narrower scope dropped it to 1.349×. That second result is the one I would chew on: letting the agent change the whole harness, not just append a note to its prompt, appears to carry real weight.
It also lets the agent decide when to stop and reflect, which fits how I would want a Claude Code style loop to behave. A fixed timer interrupts at arbitrary points. A self-triggered pause comes right after the agent notices it is stuck.
Where I would not copy it yet#
The authors call BoxBox "a research prototype rather than a production-ready self-modifying agent framework" and admit it lacks a complete security boundary. A harness that rewrites its own tools and execution logic is a harness that can rewrite its own guardrails. They add shadow verification, which runs submissions in separate containers to limit reward hacking, but that is mitigation, not a boundary.
The evaluation is also narrow. One model family, one benchmark of performance optimization where a numeric speedup gives a clean signal, and a 4-hour budget on a 96-vCPU machine. Whether self-editing helps on tasks where success is fuzzy, like feature work with vague specs, is untested here.
What I would try this week#
Not the full kernel. Three cheap moves instead:
Give the agent an explicit pause-and-review tool and log what it changes, rather than scheduling reflection on a timer.
Keep harness edits in version control inside the run so a bad edit can be reverted and a later review can read the history.
Stop spending evenings on offline prompt tuning unless the held-out numbers beat an unchanged baseline, because here they did not.
My prediction: the interesting question over the next year is not whether agents can edit their harness, but which parts they should be forbidden from touching. The 1.349× restricted-scope result says the answer is not "most of it".