← all writing
06 · 13 Nov 2026 · 6 MIN READ

Your AGENTS.md rules are switches, so test each one like code

Most of us edit AGENTS.md or CLAUDE.md the way we tune a prompt. A rule gets ignored, so we rephrase it, add IMPORTANT in caps, move it higher in the file, and hope the next run leans a bit more our way. That habit assumes compliance is a dial you can nudge.

A study from the University of Salerno and NAIST says it mostly isn't. In Do They Do as They Are Told? Investigating Coding Agents' Adherence to Explicit Instructions in Context Files, Gianmario Voria, Yutaro Kashiwa, Gemma Catolino and Fabio Palomba took real context files from public GitHub repos, turned the individual instructions into executable check scripts, and ran those checks against the commits the agents actually made. Each rule tended to be either always obeyed or never obeyed. Very few landed in between.

ACF-based governance behaves less like a dial agents turn to a partial setting, and more like a switch.

If that holds, rewording a rule that's being ignored is mostly wasted effort. You need to find out which of your rules are switched off and stop leaving them to prose.

How you turn "use pnpm" into a test

The pipeline is the part worth understanding, because it's also the part you can copy. They started from 621 cleaned context-file snapshots and 6,541 agent-authored commits across 268 repositories, covering GitHub Copilot, Claude Code and OpenAI Codex. From those files they extracted 14,902 candidate instructions and kept the 6,552 that were explicit, actionable and observable in a commit. A rule like "run the linter" is clear enough, but it leaves nothing in the repo to check, so it was dropped.

For each remaining instruction, a model generated a measurement script that reads a commit's before and after file states and returns compliant or not. 4,123 instructions (62.9%) ended up with an accepted script. Configuration and environment rules were the easiest to turn into checks at 75.6%. Security (48.5%) and management (47.7%) were the hardest. After filtering to instructions that actually had relevant commits to judge, 2,053 were scored.

Nine in ten rules sit at exactly 0% or 100%

The headline averages look fine. Mean per-instruction compliance is 59.8%, and the median is 100%. The distribution is the real story:

The obvious objection is that many instructions only had one or two applicable commits, and with a single data point a rule can only score 0% or 100%. The authors checked this. 26.3% of instructions had exactly one applicable commit. When they kept only instructions with at least five, the share of clean extremes fell from 91.7% to 86.2%. That's a small drop, and the pattern is still clearly two-sided.

So more than a third of the rules that could be measured were never followed in any commit where they applied. Rewording a rule doesn't help when the agent ignores it every single time.

Testing and process rules lose; Copilot beat Claude Code

Compliance varied significantly by category (Kruskal-Wallis, p < .001). Build-and-run rules scored highest at a 69.9% mean. Implementation details scored 63.6% and documentation 60.2%. Testing dropped to 49.2%, with a median of only 37.5%. Development process (34.0%) and architecture (38.2%) both had a median of 0%. Only one gap held up at every evidence threshold: implementation details over testing. The process and architecture numbers come from 50 and 40 instructions, so they point a direction but don't prove anything.

This matches what I've seen. "Use this import style" sticks, because the agent sees the pattern again in every file it opens. "Add a test for every bug fix" and "keep the service layer free of HTTP types" ask the agent to do extra work or hold a constraint that nothing in front of it repeats, and those are the rules that quietly drop out.

Across agents, Copilot commits complied at a 66.0% mean, Claude Code at 52.2% and Codex at 51.2%. Copilot beat Claude Code at every threshold (p < .001, Cliff's δ = 0.15). Claude Code and Codex were statistically indistinguishable. I wouldn't call that a model ranking. The comparison doesn't control for repository, language or which kinds of rules each tool's users write. Claude Code users may just write more ambitious context files.

Why I trust the shape more than the percentages

The measurement layer has real problems, and the authors say so. A manual spot check found only 71.3% of accepted scripts correctly implemented their instruction. 62.3% of the scripts returned "compliant" when run on a commit that touched no relevant files, so the applicability filter has a lot of work to do. Scripts see only final file states, not diffs, so a violation that was already in the repo can be blamed on the agent. Validation relied on one annotator. There's also no counterfactual. A rule followed 100% of the time may describe something the model would have done anyway without being told.

Those errors push in different directions. Permissive scripts inflate compliance, while scripts with hard-coded example values deflate it. I wouldn't quote 59.8% as a precise number for anything. But a two-sided split surviving noisy checks and a minimum-evidence filter is harder to explain away than a single average, so I find it fairly convincing.

Give each rule a check or move it into tooling

I'm going to change how I maintain context files. Here's the workflow this paper suggests, and it's cheap enough to start right away:

The authors frame this the same way. They describe context files as specifications that should be verified and enforced, not only written. That fits spec-driven development: if a requirement has no test, you can't tell whether it's being met.

My prediction: within a year, context-file linters that flag unverifiable rules and report per-rule compliance from recent commits will ship in the major coding agents. The open question is what a context file is for once its enforceable rules have moved into hooks. My guess is it shrinks into a short explanation of why the hooks exist, which would honestly be an improvement on most of the ones I've written.


Working on something similar?

Say hello — I read every email.