Seven papers landed in the last ten days that, read together, say something uncomfortable: most of what we attribute to the model is actually the harness. Same weights, different scaffolding, wildly different results. Meanwhile the benchmarks we quote in launch posts are measuring a workload almost nobody actually has. I build agentic coding tooling for a living, so this week's reading felt less like research and more like a code review of my own assumptions.
Same Model, Different Harness: Different Coding-Agent Results#
Sydney Lewis (arXiv:2608.26218, 26 Aug) holds the model and the task fixed and changes only the harness — what the agent sees, which tools it has, how the loop advances. The control shows the full conversation chronologically. The treatment keeps the same record but progressively compresses older tool results as context fills, and actively detects repeated or stuck work.
On a deliberately tight configuration — 169 SWE-bench Verified tasks, a 20,480-token window, a hard 480-second cutoff — mean per-task fail-to-pass fraction goes from 28% to 49%, and complete solutions from 43 to 72. That is not a tuning delta. That is the same model nearly doubling its score because someone changed what it was allowed to look at. The frozen treatment then improves three other differently-designed models with no model-specific retuning, and cuts prompt tokens per turn on the wide-window runs.
Why it matters: Every benchmark table I have ever cited silently bundles model and harness into one number. This paper is the cleanest argument I have seen that the two need separating — and that context compression is not a cost-saving hack you bolt on at the end, it is a capability lever. If you are shipping an agent and treating the scaffold as plumbing, you are leaving most of your headroom in the plumbing.
A Few Pages of Markdown: Committed AI Configuration and Lower Quality Cost#
Denisov-Blanch and colleagues (arXiv:2608.25241, 26 Aug) introduce RAMP, a four-level maturity model for the AI configuration teams actually commit to version control — behavioral rules and coding standards, then named agent definitions, then multi-agent orchestration. Across 441 repositories the levels behave as a proper cumulative scale, and human annotators reproduce the labels on 97% of a held-out sample.
The headline finding is the divergence. Agents speed everyone up roughly equally — 28–38% more commits regardless of maturity. Quality does not follow. Among agent-first repositories, those with no committed AI configuration show about twice the increase in cognitive complexity (+53% versus +27%) and 1.7x the increase in static-analysis warnings. There is also a quietly damning adoption statistic: 73.8% of these artifacts are committed once and never touched again. The authors are careful that this is observational — engineering discipline could be doing the work — and they frame it as hypothesis-generating.
Why it matters: This is the first study I have seen that puts a number on the thing every practitioner suspects: the CLAUDE.md or AGENTS.md file is not documentation theater, it is a quality control. The set-and-forget stat is the actionable half — a config written once at adoption and never revised is the norm, not the exception, and it is exactly the artifact that should evolve fastest as the codebase does.
SWE Refactor Bench: Long-Horizon, Whole-Repository Stack Migration#
Hong et al. (arXiv:2608.23564, 24 Aug) ask whether agents can carry out an entire stack migration: 20 whole-repository migrations across four technical-debt categories, scored by a three-stage protocol that separates migration completeness from behavioral correctness.
The numbers are brutal and useful. Across 520 runs from 8 frontier models, 28 runs — 5.4% — pass all three stages. The best model scores 47.0 out of 100. Thirteen of the twenty tasks received no accepted solution at all. The category spread is enormous: 31.4 on build-toolchain rewrites versus 5.6 on language rewrites. And the near-miss distribution is the part I keep thinking about — among 340 runs that passed the migration audit, 58% reached 99% of checks but only 26% reached 100%.
Why it matters: That 99%-but-not-100% cliff is the whole story of long-horizon agentic work. A migration that is 99% complete is not 99% valuable; it is a broken build someone now has to finish by hand, without having done the first 99% themselves. The paper's separation of completeness from correctness is the right primitive, and it maps directly onto how I would gate an agent on this kind of task: never on 'did it finish', always on 'does it still behave'.
Update from Hell: Hidden Breakage in Dependency Upgrades#
Luo et al. (arXiv:2608.30300, 31 Aug) target the least glamorous and most universal maintenance task there is. DEPBENCH collects 203 real-world dependency-upgrade tasks across five package ecosystems and five language communities, each one involving code-level changes that the upgrade does not announce — changed signatures, shifted types, altered runtime semantics.
The best completed configuration solves 104 of 203 tasks, or 51.2%, with wide variation across harnesses, models, and ecosystems. A coin flip on the work that fills a real maintenance backlog.
Why it matters: Dependency bumps are the task I would most like to hand off entirely, and they are the perfect adversarial case: the signal you need is precisely the thing not written in the changelog. Note again that harness variation shows up as a first-order term alongside model choice. That is now three papers in one week making the same point from different directions.
RealSWE: What Real User Requests Actually Look Like#
Kim et al. (arXiv:2608.27831, 28 Aug) do the measurement everyone skipped. They build a six-category information taxonomy, apply it to real user prompts alongside SWE-bench Verified and Pro problem statements, and quantify the gap: requests carrying only a problem statement account for 88% of real prompts but just 7% of benchmark problems. 87% of real prompts are casually written; 94% of benchmark problems are formal.
RealSWE then builds 381 task families where variants share the same underlying task and gold patch but differ in information composition and style. Realistic inputs cut resolution rates by 6.4 percentage points on average and can reorder the model rankings. The controlled analysis is the practical payload: stating Desired Behavior and Motivation significantly improves performance, while Environment Information and Reproduction Steps mostly just add tokens. Linguistic style barely matters.
Why it matters: Two things I can use tomorrow. First, benchmark leaderboards are computed on inputs that look nothing like what my users type, and the ordering itself is not stable under realistic prompts. Second, the advice we give users is backwards — we tell people to paste stack traces and repro steps, when the thing that actually moves the needle is saying what they want to happen and why. That is a prompt-template change, not a model change.
Measure Before You Manage: Agent Working Memory#
Chen et al. (arXiv:2608.31057, 31 Aug) analyze 55 archived coding-agent trajectories and find that semantically different working-memory objects — a file read, a test output, a plan — exhibit distinct retention and compression behavior. Treating them as one undifferentiated token stream throws that structure away.
They evaluate object-aware compression and retrieval-based policies, and report two failure modes that should worry anyone tuning this: improvements on training data did not reliably transfer to new tasks, and identical token allocations did not produce equivalent delivered context or equivalent management cost. Their conclusion is a four-level evaluation — stored state, delivered context, management work, task outcome — because system-level testing surfaced constraints that theoretical budgets missed.
Why it matters: Read this immediately after the harness paper. That one shows compression is worth a lot; this one shows compression is easy to overfit and hard to measure honestly. 'Same token budget' is not a controlled comparison. If you are building context management, you need to measure what actually reached the model and what the management itself cost, not just what you budgeted.
Beyond the Payload: Your Invocation Is Part of the Attack Surface#
Zhu et al. (arXiv:2608.30686, 31 Aug) look at repository poisoning from the user's side. Prior work focused on how the attacker injects and disguises a payload; this paper studies how the developer's own choices — what to delegate, how to phrase it, which skills and rules to load — change the outcome. CIPR spans 1,920 instances across 20 poisoned real-world repositories, four task types, three prompt styles, and three skill/rule conditions.
Task type alone creates up to a 4.5-fold difference in attack success rate. Test execution is the standout: high success rate, low alert rate — a silent attack surface. Prompt phrasing shifts risk indirectly rather than directly; underspecified prompts happen to reduce success by truncating how deep the agent executes, and noisy prompts trend toward suppressing alerts by making malicious content less conspicuous.
Why it matters: 'Clone this repo and run the tests' is something I do without thinking, and it is the single worst thing you can ask an agent to do in an untrusted repository. The framing I am taking away is that agent vulnerability is not a fixed property of the model or the tool — it is a function of the task you handed it. That belongs in threat models, and honestly in onboarding docs.
The Common Thread#
The harness is the experiment, not the model. Three independent papers this week — the harness A/B, the dependency benchmark, the migration benchmark — report harness variation as a first-order effect alongside model choice. A benchmark number without a harness spec is close to meaningless, and the biggest available win for most teams is scaffolding they already control.
Our benchmarks measure a workload that does not exist. RealSWE puts it starkly: the prompt shape covering 88% of real requests covers 7% of benchmark problems, and fixing that reorders the leaderboard. Meanwhile SWE Refactor Bench and DEPBENCH show that on the long-horizon maintenance work teams actually want automated, frontier agents sit between 5% and 51%.
Context is now both the capability lever and the attack surface. Compressing it well nearly doubles task success; measuring it honestly is harder than it looks; and what you let into it determines whether a poisoned repo owns your machine. The same substrate shows up in all three findings, which suggests context engineering is not a subfield of agent work — it is the field.
The through-line, if I am honest about it: the model is increasingly the part I do not control and increasingly not the part that is limiting me. Everything above is engineering — the loop, the config file, the prompt template, the trust boundary. That is a more encouraging read of the week than it first appears.