Most agent-RL papers in the morning list announce a recipe and attach a number to it. This one does the opposite: it takes a recipe everybody already uses — training a coding agent across several harnesses at once — pulls it apart into its two separate design choices, and reports honestly that one of them does nothing measurable. The interesting part is what it found while measuring. The scaffold you evaluate in matters roughly four times more than the training recipe you argue about.
What it does#
The paper is What Does Multi-Harness RL Learn? Credit Assignment and Portability in Coding Agents (Le, Cheng, He, Li, Li, Chen — submitted September 3). Its target is a recipe that has quietly become standard: run agent RL through full execution harnesses rather than a stripped-down sandbox, and use several of them. That recipe actually bundles two independent choices. First, exposing the policy to more than one harness at all. Second, pooling rewards from different harnesses into a single relative-advantage group when you compute the update. Papers report both together and credit the result to "multi-harness training." This one holds the first choice fixed and isolates the second.
The setup is deliberately boring, which is the point. One Qwen3-8B supervised warm start. The same frozen task-harness records replayed from Aider, OpenHands, Qwen Code, and SWE-agent. The same number of updates. Two GRPO grouping rules — Within, one advantage group per task-harness pair, and Cross, harnesses pooled within a task. Every checkpoint is then scored against a sealed SWE-bench Verified oracle on all four source harnesses and on a fifth minimal harness held out of training entirely. Three seeds, eight attempts per task, 24,000 sealed evaluations. There is also a probe I want to flag now because it carries the paper: an out-of-fold classifier trained to recover which harness generated a trajectory, using only the advantage signal the policy was updated on.
The key result#
Across 24,000 sealed evaluations, the evaluation harness moves the mean solve rate from 2.14% to 9.27% — a factor of 4.3. The training recipe moves it by 1.16. That single comparison is the paper. Everything else lands inside its shadow. The grouping rule, the thing the study set out to test, turns out not to matter: on the held-out harness, Cross minus Within is +0.25 pp with a 95% confidence interval of [-0.48, +1.02] at eight attempts per task, and +0.16 [-0.41, +0.72] pooled across three training seeds whose individual estimates change sign. Each rule's own seed-to-seed range, 0.42 to 0.45 pp, is wider than the gap between the rules. Both put their largest gains on the same source harness. The classifier probe explains why: harness identity is recoverable from Cross's pooled advantage at +4.48 pp above a shuffled-label baseline, and from Within's not at all — yet the two rules still reach the same held-out score and the same action distribution inside each harness. Re-collecting half the training data on-policy does not change any of this.
Why it matters#
Read the 4.3× first, because it is a fact about how you should read every other agentic coding number you saw this year. Every model-versus-model comparison, every fine-tune ablation, every "our agent scores X on SWE-bench" was measured inside some scaffold, and this paper says the scaffold is a bigger effect than the thing being compared — by almost four times. The practical consequence is unglamorous and immediate: pin your harness, version it, and report it next to the number, the same way you'd report the model ID. When you read someone else's result, treat the harness as part of the claim rather than as plumbing. And when you're choosing between models for a specific product — a Claude Code sub-agent, a homegrown loop, a CI bot — measure inside your scaffold, because a delta observed in someone else's may not survive the move. The flip side is a cheap win most teams under-invest in: if the harness is the dominant variable, then tool surface, context assembly, and retry policy are higher-leverage work than another round of fine-tuning.
The Cross-versus-Within null is the deeper finding, and it's aimed at anyone actually training agents. The intuition behind pooling rewards across harnesses is appealing — comparing a task's outcomes across scaffolds should factor out scaffold-specific noise and teach the model what is common to solving the problem. The classifier probe says the opposite happened. The pooled advantage encodes which harness you are in, not what solving the task requires. That's configuration adaptation dressed up as generalization, and it produces exactly as much portable capability as the simpler rule: none extra. So the concrete change is twofold. Stop reaching for multi-harness pooling as a generalization lever on the assumption that it transfers — that assumption now has evidence against it. And put a genuinely unseen harness in your eval suite before you claim portability, because the four harnesses you trained on cannot tell you whether you learned the task or learned the scaffold.
The caveats#
One backbone, one warm start. Everything runs from a single Qwen3-8B SFT checkpoint. A null at 8B is not a null at frontier scale — it's plausible that a bigger model has the capacity to extract the shared structure that pooling is supposed to expose, and this study cannot see that.
Mostly replayed, not on-policy. The main comparison replays frozen task-harness records. The authors check this by re-collecting half the data on-policy and the conclusion holds, which is the right test — but it's half, not all.
One held-out harness. Portability is measured against a single "minimal" scaffold. That is one sample of the space of unseen harnesses, and it happens to be the least featureful one.
A null is a null. The confidence intervals still admit roughly a point of upside for Cross. The claim is that the effect is smaller than seed noise, not that it is exactly zero — and SWE-bench Verified carries its own well-documented baggage as the oracle.
The takeaway#
What I'm filing away is the reframing: the harness is a first-class experimental variable, not infrastructure that sits underneath the experiment. A 4.3× versus 1.16× split means most published agent comparisons are partly measuring their scaffolds, including mine. Concretely, two changes. I'm going to pin and report the scaffold version alongside every agent eval number I produce, so a result stays interpretable a month later. And for any change I want to claim generalizes, I'm adding one deliberately unfamiliar harness to the suite — not because it flatters the number, but because it's the only part of the eval that can tell the difference between a better agent and a better-adapted one.
Paper: arxiv.org/abs/2609.04518 — Chenqian Le, Jiayi Cheng, Qijia He, Runhao Li, Yinghao Li, Xupeng Chen. Submitted 3 September 2026.