Most mornings the arxiv list is another variation on "we built an agent, here is a benchmark it does well on." Today one paper inverted the setup and made me stop scrolling. Instead of asking how well a model solves tasks inside a harness, it asks whether the model can build the harness itself — and then whether it can tell if its own edits actually helped. The first half mostly works. The second half falls apart, and that is the part worth your attention.
What it does#
HarnessDev (Wu, Zhang, Shi et al., 19 authors, submitted September 1) moves the unit of evaluation from task outputs to runnable infrastructure. The premise is one every practitioner already lives with: hold the weights fixed, change the harness, and downstream performance moves a lot. Yet we evaluate models under a harness someone else chose and report the number as if it were a property of the model. HarnessDev makes the harness the thing under test.
It runs in two stages. In Creation, the model starts from a deliberately weak seed — a stable CLI, runtime model config, audit writers, and no task-solving policy whatsoever, scoring a flat 0.0 on every benchmark — and has to build a complete execution system implementing six modules: execution, tools, context, state, lifecycle, and evaluation. In Evolution, it takes its own created harness and iteratively revises it using downstream execution feedback. Six creator models (Opus 4.8, GPT-5.5, Gemini 3.1 Pro, DeepSeek V4 Pro, Qwen 3.7 Max, Seed 2.0 Pro) build harnesses across four domains and five benchmarks — SWE-bench Pro, Terminal-Bench 2.1, MLE-bench, EQ-Bench3, BrowseComp — totalling 2,207 downstream instances, with hidden evaluation tasks withheld during development. Crucially, every finished harness is frozen and then run twice: once under the model that built it (Self-Eval) and once under a fixed executor (Unified-Eval). That second run is where the paper earns its keep.
The key result#
On Creation, the headline is a respectable near-miss: Opus 4.8 is the strongest builder at 67.8 average versus 86.2 for human-engineered references, with writing harnesses essentially matching the reference and search harnesses collapsing (Opus scores 3.1 on BrowseComp against a reference of 92.2). But the number I keep coming back to is from Evolution. All five self-runtime creators improved on the feedback set they could see — gains from +3.0 to +13.9 points. On held-out tasks those gains shrank to +1.43 to +4.44. And across 64 comparable version switches, the feedback score and the held-out score moved in the same direction only 34 times — 53.1%. Only 2 of 9 versions the loops declared best were actually the held-out optimum. The signal these self-improvement loops optimize against is, for selection purposes, a coin flip.
Why it matters#
Start with the transfer result, because it has an immediate operational consequence. When Opus's own SWE-bench Pro harness was handed to Gemini as the executor, its score fell from 69.3 to 33.0. Qwen's harness went the other way, gaining 17.6 points on BrowseComp and 12.9 on MLE-bench under Gemini, because its original executor was the bottleneck all along. What the model built was not a general execution system; it was a compatibility layer co-adapted to one specific executor. If you have accumulated a CLAUDE.md, a set of skills, sub-agent definitions, and MCP wiring, you have built a harness — and this paper is direct evidence that a chunk of its value is model-specific. The practical move is unglamorous: when you bump the model, re-run your eval set rather than assuming the scaffolding carries over, and treat a harness tuned against one model as a hypothesis about the next one, not a result.
Then there is the eval-design half, which I think is the more important lesson. Any loop where an agent edits its own prompts, skills, or tool definitions and keeps the edits that improve a measured score is doing exactly what HarnessDev's Evolution stage does — and will inherit its failure mode. Feedback-set gains that are three to four times larger than held-out gains is the signature of a loop fitting its own scoreboard. The fix is boring and well known, which is precisely why it gets skipped: keep a held-out slice the tuning loop never touches, evaluate the whole lineage after freezing instead of trusting the version the loop nominates, and treat visible feedback as a local search signal rather than a selection criterion. The cost finding reinforces the point — token usage on MLE-bench varied roughly nineteen-fold across harnesses with no reliable relationship to score, so "it works" and "it works at a sane price" are separate questions you have to measure separately.
The caveats#
Evolution has one trajectory per creator–runtime cell, so there are no uncertainty estimates. The 53.1% is directional evidence, not a confidence interval.
The post-freeze held-out evaluation covers SWE-bench Pro only, and Evolution focuses on code harnesses. The generalization claim is narrower than the framing suggests.
The human baselines are uneven and not optimized, and three of the reference values are taken from vendor release reports rather than local reruns under matched conditions — so the "gap to human harness" is softer than a single number implies.
Unified-Eval reduces executor-model differences but cannot remove them; harness–model interaction is not cleanly separable.
The development environment is held fixed across both stages, so whether an evolved harness can bootstrap its own further evolution is untested.
The takeaway#
What I am filing away is a single distinction: the set that generates your feedback and the set that selects your final version must not be the same set. That sounds obvious written down, and it is exactly the discipline that erodes the moment you have an agent loop that reports a rising number. Concretely, I am doing two things differently. First, whenever I change the model behind a project, I re-run the evals instead of assuming the accumulated config still earns its keep. Second, for anything that self-tunes, I am keeping a held-out slice the loop can never read, and judging the whole lineage against it after the fact rather than believing the version the loop declares best. HarnessDev's real contribution is not that models can build harnesses — it is that they cannot yet tell which harness they should keep.
HarnessDev: Can LLMs Create and Evolve Their Own Agent Harness? (arXiv:2609.01437) — Yuhao Wu, Jingyuan Zhang, Jiajun Shi, Xinping Lei, Qingshui Gu, Yuxuan Zhang et al. cs.SE, cs.CL. Submitted September 1, 2026.