64 of 80 runs. That is how many times a Claude Code agent, working in a repo whose AGENTS.md pointed at a test command that no longer existed, made at least one call that ran no tests and raised no error. Task success in that condition was 79 out of 80, the same as with the corrected file. If you only look at pass rates, this paper's finding is invisible.
The study is "When Instruction Files Go Stale: Documentation Debt and Hidden Cost in Coding Agents", from a team at NC State. It is small and exploratory, and the authors say so. But the thing it measures is something I think most of us never look at.
Success stayed flat while the trajectories did not#
The setup: three real stale instructions in three open-source repos, each a different kind of guidance. A test command (pyflow, tests had moved), a code location (cog, an interface file had moved) and a procedure for adding a component (aghub). 22 small maintenance tasks, each a bug caught by the repo's own tests. Every instruction came in three versions: stale, corrected, removed. Two models (Haiku 5.5 and Sonnet 5.5) in Claude Code, 732 runs, analysed down to individual tool calls.
The headline contrast is stale-path use under the stale condition:
Command case: 78 of 80 runs followed the stale command
Procedure case: 55 of 80
Location case: 0 of 84, because no task needed that file
Corrected condition: zero stale-path use in every case
So agents do follow the line, but only when the task routes through what it names and nothing else offers a starting point. A stale pointer to code nobody needs is harmless. A stale command that every task needs is not.
The expensive failure is the one that looks like a pass#
The cost numbers are modest, and the paper flags them as exploratory. On the test command, Sonnet averaged 7.8 tool calls per task with the stale line versus 5.8 with the corrected one, roughly a third more. Tokens were 195k versus 152k, which was not significant. No other case showed a resolved difference in calls or tokens.
What bothered me more is what the stale command did when it ran. In 64 of 80 stale Command runs, at least one stale-path call executed no tests, and none was flagged as an error. In 19 of Sonnet's 40 stale runs, the very first check of the fix ran no tests at all, versus 0 of 40 with the corrected file. A green exit code from a runner that collected nothing is a verification step that verified nothing.
This lines up with a pattern from other papers I have covered: an agent's own report that it checked its work is weak evidence. Here the mechanism is mundane. The harness does not lie, the instruction file does, and the agent has no reason to distrust it.
Agents recover, and then leave the mess#
Recovery was good. Agents reached the current target in 130 of 133 stale-path encounters, 98 of them via repository search, with a median of about two extra calls. That is why success stayed flat.
The part that matters for maintainers:
none repaired the underlying instruction file
Only 21 of 133 encounters had a final response that identified the mismatch. Agents cited the instruction file as the source in just 2 of 133 runs, so the stale path read like ordinary knowledge, not like something to question. And no run edited an instruction file or proposed to. Every future session pays the same toll. Deleting the line removed most stale-path use, but agents still guessed the old folder in some runs (20 of 80 in the Procedure case).
What I would change this week#
The authors suggest safeguards but did not evaluate them, so treat these as my reading, not their result:
Make "no tests collected" a failure in whatever wrapper the agent calls, instead of trusting the runner's exit code.
Put a cheap check in CI that every path and command named in AGENTS.md or CLAUDE.md still resolves.
Add one line to the instruction file telling the agent to report and fix a mismatch it finds. Nothing in the study shows agents do this unprompted.
I would also steal the evaluation lesson. If you build evals for coding agents, log what the agent did, not only whether the tests passed. A suite that only scores outcomes would have rated all three instruction conditions as equivalent.
There is a design point underneath this. An instruction file is read in every session, so its errors compound the way a bad shared helper does, except nobody reviews it like code. It has no tests, no owner and no failing build when it rots. The agent, meanwhile, is optimised to get the task done, so it routes around the damage quietly. Both sides behave sensibly, and the debt still grows. That is a process gap, not a model gap, and it is why I would fix it with tooling instead of hoping a better model notices.
How far to trust this#
Three cases means instruction type, repo and task vary together, so you cannot attribute the effect to any one of them. Most tasks are small seeded defects, and harder maintenance work might show bigger costs or different recovery. Only two models from one vendor, in one harness. The authors also found and fixed a detector bug mid-analysis that changed 19 Sonnet runs in the Procedure case, which is honest of them and a reason to hold the exact counts loosely.
My prediction: instruction-file linting becomes a standard CI step within a year, the same way we lint dead links in docs. The open question is whether agents should be trusted to patch their own instructions, and nobody has measured that yet.