Most of this morning's agentic-coding arxiv drop was benchmarks. This one wasn't. A group from MPI-SWS, EPFL, and Aarhus took Denning's 1968 working-set idea from virtual memory and pointed it at coding agents, then did the thing benchmark papers rarely do: they supplied and withheld the required facts themselves, one at a time, and watched what broke. The finding that stuck with me is that an agent missing a fact it needs usually doesn't say so. It writes the file anyway.
What it does#
The paper models a repository-scale task as a coupled-fact graph. Nodes are atomic facts — symbols, tests, configuration values, imports, migration rules, invariants. An edge means that changing one endpoint requires the other to stay consistent. At each edit, the facts that edit needs arrive through one of two channels: recent context (the agent read it, or the harness supplied it) or parametric memory (the model already knows it). Whatever neither channel covers is what The Working Set of a Coding Agent: Coherence Debt in Repository-Scale Tasks calls coherence debt. The claim baked into that definition is substitutability: a fact should count the same whether it came from a file read or from pretraining.
What separates this from prior repo-level work is that they manipulate both channels rather than observing them. Four fictional API migrations, each with 12 mechanically checked requirements, run closed-book — no workspace, no tools, both channels empty. A matched front-loaded condition puts the exact rules and sources in the prompt. A real Pydantic v1 to v2 migration with 79 tests runs alongside a twin with every API name renamed, holding the task fixed while making memorized knowledge useless. Then synthetic tasks where the authors wrote the coupling themselves, so they know precisely which facts each edit requires. Seven models across five harnesses: Claude Code, Codex CLI, Aider, OpenHands, and a shell-only probe.
The key result#
The withhold sweep is almost boringly clean, and that is the point. Withholding k of eight required facts and rerunning five configurations at k in {0, 4, 8}, every configuration scores 100%, roughly 50%, then 0% — no two differing by more than three points at any level. Withholding a fact costs exactly the work it supports and no more. Distance doesn't matter either: a supplied fact is used as reliably 128,000 characters from the edit as sitting beside it. What varies wildly is the bill. Across 144 trials that all pass every test, peak per-turn context differs by only 1.8×, but cumulative input tokens differ by 12.8×, from 293,882 to 3,752,134. Same working set, very different rebuild rates — five tool calls at the cheap end, seventy-nine at the expensive one. Haiku burns 5,730,807 input tokens to reach the same 100% that Opus reaches on 459,122. And when facts are withheld, the expensive configurations recover nothing.
Why it matters#
The fabrication result changes how I would instrument an agent. Withhold a required file and see who mentions it: Opus in Claude Code reports the missing file in all 8 trials, Fable 75%, Sonnet 25%, Haiku 12.5%, while GPT-5 in Codex CLI and GLM-5.2 in opencode report it zero times and produce a confident wrong migration instead. Pooled across 46 trials, 37%. Within a model the rate is stable, which makes this a selection criterion rather than a prompt you can patch. It also breaks read-based instrumentation at the root: because the agent answers a missing fact by acting, the shortfall enters the tool stream as more activity, not less. The authors found their own read-derived coverage score was crediting files the agent had written itself — returning a perfect 1.000 in the condition where nothing was available, and 0.000 once agent-authored paths were disqualified from the numerator. If you compute context-coverage metrics from tool events, add that exclusion.
The convention-file finding is the one I am acting on today. On contested surfaces — integer cents versus float division, .get() with a default versus direct indexing, timezone-aware versus naive timestamps — agents follow a written standard over working code in all 39 trials, across two harnesses and three model tiers. The authors then inverted the workload so the document demanded the worse practice, and the agents still followed the document: camel-case handlers, money divided into floats, the naive clock. Scored over 3,385 decisions, a correct standard yields 100% of decisions written the better way, code alone 33%, and a standard demanding the worse form 0%. A stale CLAUDE.md is worse than no CLAUDE.md, because it suppresses the inference the agent would otherwise have drawn from the code — which lands hard next to the catastrophic-remembering result on ever-growing context files I covered a week ago. The decomposition cells point the same direction: sub-agents were safe on independent fixes (3/3 in both modes, and 30% faster) and harmful on a tightly coupled migration (Codex 2/6 decomposed against 3/3 single-worker). Partition on coupling, not on convenience.
The caveats#
Six of the seven main workloads are migration-shaped, where a latent graph has to be discovered. Greenfield work builds its graph while writing and may behave differently — the authors say so explicitly.
Several of the most quotable cells are tiny. The decomposition table runs 3 to 6 trials per cell, and the paper labels those directional observations rather than results.
The read-based residency proxy is coarse: import edges recover 40% of the authored coupling at 40% precision, and a four-feature predictor drops from 0.71 held-out AUC to 0.66 under leave-one-workload-out.
Most usefully, the framework fails where they say it fails. On SWE-bench Verified the final-quarter residency score gives within-cell AUC of about 0.49 — chance. An earlier version scored above 0.92 through an outcome-gated term that leaked the label, and they report that. They also withdraw their own earlier lean-versus-bloat reading after rerunning it at twenty trials per condition. That kind of self-correction is rarer than the headline result.
The takeaway#
What I'm filing away: coverage at write time is the variable, and nearly everything we reach for instead — bigger context windows, more tokens, more turns — is a proxy that can be fully satisfied without satisfying it. Two changes on my side. I'm auditing convention files for staleness on the assumption that a wrong one is strictly worse than a deleted one, and I'm going to stop treating "the agent read the file" as evidence it has the fact. The check this paper argues for is against what the agent produces, not what it opened — and given how rarely most configurations announce that they're stuck, that is the only signal not already contaminated by the failure it's meant to detect.