Six papers landed on arxiv in the last ten days that are worth your time, and what struck me reading them back to back is how few of them are about the model. One measures how agent teams talk to each other. One asks whether a spec written for one agent still means anything to another. One is about the fact that retrieval has no concept of time. The frontier this week is not capability — it is the scaffolding, the specs, and the memory we wrap around capability, and how badly we are currently measuring all three.
Specification Portability Across LLM Development Agents#
Grynets et al. (arXiv:2608.21208, August 21) take a question most of us have quietly assumed away — is a spec a portable artifact? — and answer it with an Oracle-to-PostgreSQL migration harness. Stage one runs a spec-first pipeline over 1,006 PL/SQL files: 623 were successfully regenerated, and 380 of those scripts actually executed in PostgreSQL 16. Stage two is the interesting half: 1,802 Oracle scripts with known PostgreSQL implementations, run across Amazon Kiro, Google Gemini, and GitHub Copilot, with Claude Code and Cursor in the single-agent evaluation. Specs are graded native (written by the agent executing them) against foreign (written by a different agent) on Token F1, exact match, SQL syntax validity, AST exact match, AST mean similarity, and immediate runnability.
The degradation is not subtle. The strongest replicated failure came when Gemini directly consumed a Kiro-origin specification: Token F1 of 0.035, SQL syntax validity of 2.33%, AST mean similarity of 0.015. That is not a lossy handoff, that is a total one. Two secondary findings are just as useful: specification size alone does not predict implementation quality, so the instinct to write longer specs is unfounded; and compression did not deliver a universal benefit. Rewriting a foreign spec into the target agent's idiom helped Gemini substantially, and retrieval-augmented ingestion was the only strategy that appeared on the per-agent Pareto frontiers of both Gemini and Copilot.
Why it matters: A lot of teams have converged on a workflow where planning happens in one tool and execution happens in another — write the spec in a chat model, hand it to a coding agent. This paper says that handoff carries an unmeasured tax, and that the tax is agent-dependent rather than uniform. If you run a heterogeneous SDD setup, the spec is not a neutral document you can pass around; it is an artifact tuned to whoever wrote it. Retrieval-based access to the spec, rather than dumping it into context, is the one mitigation that held up across two different agents.
BC-Bench: Agentic Engineering in an ERP Domain-Specific Language#
Sun and Hansen (arXiv:2608.20851, August 21) built a benchmark for AL, the DSL behind Microsoft Dynamics 365 Business Central — 101 manually curated tasks pulled from two Microsoft-owned production repositories. They adapt the SWE-Bench methodology to an ecosystem with almost no public training material and genuinely painful environment provisioning, and they go beyond patch generation to evaluate test generation and support multimodal problem statements, since visual context turns up constantly in real ERP tickets.
The headline result is the one worth arguing about: in the bug-fixing category, differences between models in resolution rate were larger than differences between the two agent harnesses evaluated. And critically, improvements reported on general-purpose benchmarks did not consistently transfer to AL. A model that climbed the public leaderboard did not reliably climb BC-Bench.
Why it matters: This is the cleanest evidence I have seen that a SWE-bench number is a measurement of one thing — can this system do Python-shaped work on well-represented open-source repos — and not a general capability score. If your codebase is a DSL, a proprietary framework, or anything thin on public training data, the public leaderboard is close to uninformative for you and you need your own eval set. The harness-versus-model finding also cuts against the current consensus that scaffolding dominates; on this domain it did not, which is a useful reminder that the answer is domain-specific rather than universal.
Temporal Validity on Real Software Histories#
Neeraj Yadav (arXiv:2608.20685, August 21) names a failure mode I have hit repeatedly and never had a number for. Retrieval-augmented generation has no model of time. When a fact changes mid-session — a function gets renamed, an endpoint moves, a dependency is bumped — RAG retrieves the old and the new value with near-identical similarity and has no way to tell which one is current. So it serves the superseded value.
The validation is done on real history rather than synthetic pairs. From 707 real GitHub issues (SWE-bench Lite plus Verified) the author extracts 130 clean atomic state transitions — a fix that changes exactly one identifiable value — and renders each marker-free, so the stale and current statements differ only in the value itself. MemStrata, a deterministic (subject, relation, object) supersession memory, reaches 0.91 answer accuracy against RAG's 0.57–0.59. The structural result is sharper than the accuracy gap: when forced to answer, RAG serves the superseded value 36–38% of the time, an LLM reranker does not help, and MemStrata drives that to roughly zero — at RAG's retrieval latency, about 2.1 seconds versus roughly 18 seconds for the reranker path. The paper is unusually honest about scope: only about 18% of real fixes are clean atomic transitions, and extraction coverage for the rest is explicitly deferred.
Why it matters: Every agent product now ships something called memory, and most of them are similarity search over an append-only log. That architecture is a stale-fact generator. 36–38% is the number to carry around: on the class of change where you can cleanly identify what was superseded, naive retrieval gets it wrong more than a third of the time, and throwing a reranker at it does not save you. Supersession has to be modelled explicitly in the data structure, not inferred at read time.
ARQ: Agentic CodeQL Query Refinement#
Wang et al. (arXiv:2608.20637, August 21) point an agent at a target I had not considered: not the code, but the static analyzer that checks the code. CodeQL queries encode vulnerable patterns and match them against source, and they suffer from both false positives and false negatives. ARQ synthesizes C/C++ programs and uses execution as ground truth — if a program is genuinely vulnerable and the query stays silent, that is a false-negative weakness; if the program is safe and the query fires, that is a false-positive weakness. Those disagreements feed an LLM refinement loop that repairs the query.
The setup requires no labeled datasets, no commit history, and no vulnerability-specific templates, which is what makes it interesting as a pattern. Refining 12 official CodeQL queries with GPT-5.4, Claude-Sonnet-4.6, and Gemini-3.5-flash, the refined queries detect up to 119.8% more true positives on Juliet v1.3 and FormAI v2 while holding precision at 98.0% or above. The field validation is the part that lands: ARQ resolved three open issues in the official CodeQL query repository, one of which had been sitting there for 27 months, and the refined queries surfaced two previously undiscovered bugs in libpng and zlib.
Why it matters: The generalizable idea here is execution-as-oracle for tool improvement. Most agentic security work tries to get the agent to find the bug directly. ARQ instead has the agent improve the deterministic detector, which means the output is a reviewable artifact that runs at static-analysis speed forever after, rather than an LLM you have to re-run and re-trust on every scan. Anywhere you have a rule-based checker and the ability to synthesize inputs that settle disagreements, this loop applies — linters, type stubs, CI policies.
Terminal Agents: A Survey of AI Agents in Command-Line Environments#
Bin et al. (arXiv:2608.20485, August 20) argue that the literature has been slicing this wrong. Terminal-mediated agent behavior is currently scattered across software engineering, tool use, and computer-use research, and the survey proposes treating terminal agents as their own class: systems whose dominant progress-bearing action–observation loop runs through terminal command execution, textual feedback, and stateful environment interaction. They organize the space with a seven-dimensional terminal competence profile connecting system architecture, competence acquisition, and evaluation.
The synthesis conclusion is that realized behavior is jointly shaped by model, interface, harness, runtime, and environment — and that this joint shaping is precisely what current evaluation obscures. Prevailing benchmarks emphasize final outcomes, which exposes process quality, recovery, and governance unevenly. Their bounded fixed-condition diagnostics show two concrete consequences: different benchmark families surface different process signals, and matched system comparisons produce benchmark-dependent results that limit how much you can attribute to any single component.
Why it matters: Read alongside BC-Bench, this is a direct warning about a claim I hear constantly — 'the harness matters more than the model.' The survey's position is that component attribution is itself benchmark-dependent, so that claim is not stable across settings. The practical ask is one I would like to see adopted: report system and runtime conditions explicitly, and publish replayable traces so process-level evidence exists at all. Right now almost every agent comparison I read is a pass rate with an unspecified runtime underneath it.
When Agents Coordinate: Measuring Coordination in Multi-Agent AI Coding#
Destefanis and Aste (arXiv:2608.16801, August 17) do the empirical work on a question multi-agent frameworks usually answer by assertion: how do teams of coding agents actually coordinate? They represent each run as a temporal network — agents and files as nodes, interactions as timestamped edges — across 1,902 runs spanning different team configurations.
Direct messaging between agents grows close to quadratically with team size before levelling off in larger teams, which is the communication overhead you would expect and a real argument against naively adding agents. The mitigation is concrete: routing coordination through shared files instead of messages cut output tokens by about 42% at eight agents on message-heavy work. Task structure turns out to drive network shape more than configuration does — shared specifications produce dense, highly-connected teams, while pipeline-structured tasks produce sparse ones. There is also a finding buried in a sealed-environment test of 244 additional runs that deserves more attention than it will get: agents accessed marked placeholder files in four fifths of runs.
Why it matters: The 42% token reduction is the immediately actionable result — if you are running multi-agent setups, a shared scratch file or artifact directory is cheaper than inter-agent chat, and the gap widens as you add agents. But the placeholder finding is the one I would act on first. Agents wandered into files that were sitting there as bait in four out of five runs, which tells you that filesystem scope in a multi-agent workspace is not a theoretical concern. If your agents share a working directory, they will read things you did not intend them to read.
The Common Thread#
The interesting variance has moved out of the model. Spec provenance, memory structure, file-versus-message coordination, runtime conditions — these papers keep finding that what determines outcomes is the layer around the model. But note that BC-Bench found the opposite on its domain, with model choice dominating harness choice, and the Terminal Agents survey argues component attribution is benchmark-dependent by nature. 'Scaffolding matters more' is a finding, not a law.
Evaluation is the actual bottleneck. Three of these six papers exist because existing benchmarks did not answer the question. BC-Bench had to be built because general-purpose scores do not transfer. The Terminal Agents survey's central complaint is that outcome-only evaluation hides process, recovery, and governance. Yadav had to extract 130 transitions from 707 real issues because no benchmark measured temporal validity at all. We are still measuring the wrong things and mostly finding out after we ship.
Time and state are the unsolved primitives. Retrieval serves superseded values a third of the time. Specs decay across agent boundaries. Coordination graphs shift as tasks progress. Every one of these is a failure to represent that things change — and the two systems that fixed it, MemStrata and ARQ, both did so by adding something deterministic underneath the model rather than asking the model to reason its way out.