Most mornings the agentic-coding arxiv list is another round of benchmarks telling me which model is two points better than which on the same saturated suite. Today one paper broke the pattern by going after something I actually fight with every single week: the repository guidance file — the AGENTS.md, the CLAUDE.md, the hand-curated "here's how this repo works" note we all write and quietly distrust. It does two unfashionable things at once: it automates that file, and it dissects what the file actually buys you. My thesis: repo guidance is model calibration, not documentation — and that reframing should change how you maintain it.
What it does#
Probe-and-Refine Tuning of Repository Guidance for Coding Agents (Asa Shepard and Jeannie Albrecht) tunes a repo's guidance file with a loop that never runs an agent. No multi-step agent rollout, no RL, no gradients, no tool use during tuning — just single-shot LLM calls. Each iteration generates a batch of ten synthetic bug-fix probes from the repository's own code at high temperature, attempts a patch for each given the current guidance, judges each attempt as strong / partial / missing while proposing per-probe edits, then aggregates the diagnostics and applies at most five edits mechanically. Three to five iterations, roughly 22 single-shot calls each, and you have a guidance file specialized to that repo. The probes are self-generated and held separate from the eval instances, so there's no contamination.
That's the key departure from prior AGENTS.md work, most of which measures whether hand-written context files help. This paper instead generates and iteratively repairs the file cheaply, and compares against two honest baselines: an unguided agent, and a "static knowledge base" — a tree-sitter structural summary of hubs, entry points, and imports plus generic best-practice advice ("reproduce the failure before editing"). Probe-and-refine starts from that static base and specializes it.
The key result#
On SWE-bench Verified with a constrained Qwen3.5-35B agent (four trials, 200 steps), probe-refined guidance resolves 33.0% of issues, versus 28.3% for the static knowledge base and 25.5% unguided — an odds ratio of 2.11 over unguided, p < 0.001. But the number that actually stuck with me is what's not moving. The entire gain is coverage, not precision. Coverage — the fraction of instances where the agent emits a patch the harness can even evaluate — jumps from 41.7% to 56.2% (+14.5 points). Conditional on producing an evaluable patch, all three conditions resolve at the same ~59% rate (χ²=4.26, p=0.119). The guidance file doesn't make the agent write better fixes; it makes the agent reach the right file and emit a well-formed patch at all.
Why it matters#
This quietly corrects a belief I've been operating on. We write CLAUDE.md hoping it makes the agent smarter — better reasoning, better fixes. The data says no: once the agent gets to an evaluable patch, guided and unguided perform identically. What guidance buys is navigation and well-formedness. It cuts the agent's fallback rate from 25.6% to 14.8%, meaning the agent more often stays in its main tool-use loop instead of bailing to a degraded single-shot path. For a builder that's a precise prescription: your repo guidance should be a map — entry points, module hubs, where things live, how to scope a change — far more than a style lecture. It tracks the content the tuner actually added (47% procedural, 30% structural, 23% quality-gate): operational, not aspirational.
Then comes the kicker, and it's the part I'll be repeating to people. Guidance is model-specific behavioral calibration, not transferable knowledge. Apply Qwen-tuned guidance to a different model (Nemotron) and resolve rate collapses to 13.2% — the model lapses into "compliance by analysis," writing prose about what it would do instead of issuing edits, and fallback patches explode from 3 to 324. Even self-tuned, guidance actively hurts Nemotron (no-context 28.4% beats probe-refined 27.0%). So the AGENTS.md you crafted while pairing with one model may degrade a different model in CI, or a teammate on a different agent. If you run multi-model fleets or swap models, version the guidance per model and re-tune when you switch — and the automated loop is what makes that realistic, since re-tuning is ~22 cheap calls, not a human afternoon.
The caveats#
One model, really. The method is demonstrated on Qwen3.5-35B; Nemotron mostly supplies the negative-transfer result, not a second success story. We don't yet know it generalizes as a recipe.
Length confound. Refined guidance is 63% longer (2,754 vs 1,687 chars). Content analysis helps, but you can't fully separate "better content" from "more tokens."
Benchmark skew. Django is 46% of the 500 instances; the non-Django subset is 11 repos with one or two instances each.
Low absolute ceiling. Resolve rates of 22–35% come from a 35B MoE with 16k effective context and aggressive output truncation. On a frontier model with long context, well-formedness is rarely the bottleneck — so the coverage story may shrink where it matters most to me.
The takeaway#
What I'm filing away: stop treating the repo guidance file as documentation and start treating it as a model-specific calibration artifact that mostly buys coverage. Concretely, when an agent flails on my repo I'll first ask whether it's failing to produce an evaluable patch or producing a wrong one — because guidance only fixes the former. And I'll stop assuming the CLAUDE.md that works with one model is portable to the next; if the results matter, tune it — now cheaply, now automatable — for the model that will actually read it.