Most mornings the agentic-coding slice of arxiv is benchmarks and scaffolds. Today something different scrolled past: a paper that doesn't evaluate agents at all. It audits the configuration we write for them — instruction files, skills, hooks, MCP server declarations, subagent definitions. The stuff sitting in .claude/ and .cursor/ across thousands of public repos. Nobody had treated that layer as a supply chain before. It is one, and it is unlocked. The thesis: your harness is a dependency graph with no lockfile, and roughly one in six public setups already carries a security defect.
What it does#
Scanning the Harness: An Empirical Study of Supply-Chain Defects in AI Coding-Agent Configurations (Kapner, Soceanu, Petrunin, Gartner) studies 3,171 public GitHub repositories that ship coding-agent configuration: 2,660 “setups” that assemble two or more component types, plus 511 published skill collections. The corpus was assembled from assistant-specific topic queries (claude-code, agent-skills, agents-md, cursor-rules, copilot-instructions, subagents), README references to canonical context filenames, thirty curated lists of agent configs, and the Claude Code plugin marketplaces.
The framing is the real contribution. The harness is a dependency layer: installed from marketplaces and public repositories, running with the developer's full privileges, with no lockfile, no install-time check, and — the part that stings — no vocabulary for what a component is even permitted to do. npm has all of that. Your .claude/ directory has none of it.
The method is stricter than the usual “we ran a scanner and counted hits.” The authors define 31 rules — 15 security, 12 “cannot-work,” 2 Agent Skills spec violations, 2 cross-assistant inconsistencies — and only count rules decidable from bytes whose consequence is a security exposure, a configuration that cannot function, or a documented spec departure. Six “gating” rules clear their agreement bar and produce every headline number. Each finding then survives three passes: an independent implementation re-derives it from the repository at its pinned commit, a language-model adjudicator with a released prompt rules on disagreements, and a second independent model session re-checks every counted pair. The instrument, corpus manifest, prompt, and every verdict are released.
The key result#
16.0% of setups carry a confirmed security defect; 16.7% carry a confirmed defect of any kind. Three security classes survive validation. First, 9.8% of setups declare at least one MCP server with no version pinned — typically npx -y @scope/server-name, refetched on every start, running with the agent's privileges. Second, 3.1% pre-approve arbitrary command execution behind a grant that reads narrowly scoped, the canonical example being Bash(python:*) — because python -c runs anything. Third, 3.8% ship a skill that pre-approves the shell for whoever installs it, a class that also appears in 3.7% of published skill collections, which means a marketplace scan could catch it before install. The methodological number is nearly as interesting as the security ones: the raw scanner reported 25.5% on the identical rules. About a third of what it flagged did not survive validation. No credential-exfiltration path was confirmed.
Why it matters#
The Bash(python:*) finding is the one to internalize, because it is a design flaw in how we all reason about permissions. An allowlist gives the feeling of least privilege while granting an interpreter, and an interpreter is a universal escape hatch. Anyone who has written Bash(npm:*), Bash(make:*), or Bash(uv:*) has done the same thing to themselves. The mental model has to shift from “which binary is this” to “what can this binary reach.” The paper also surfaces a genuinely supply-chain-shaped detail: Claude Code stopped honoring permissions.defaultMode: bypassPermissions from project and local settings as of v2.1.257, but every earlier client still honors it. So a repo shipping that line is not a defect on your machine and is a live exposure on someone else's — the classic shape of a dependency problem, where the blast radius depends on the consumer's version, not the publisher's.
The second-order point is distribution. Skills, subagents, and plugin bundles are now installed the way packages are, from marketplaces, by people who read the README and not the permission block. Four things a builder should do differently after reading this. Pin MCP server versions the way you pin production dependencies — @scope/server@1.4.2, not a bare npx -y. Grep your permission allowlists for interpreters and build tools before committing them, and prefer explicit argument-scoped grants over tool:*. If you publish skills, treat a bundled shell allow as a security-relevant, breaking change that belongs in a changelog rather than quietly inside a bundle. And if you install third-party harness components, read the permission block first: 3.7% is a small number, but a pre-approved shell is effectively a bearer token for your laptop, and the install flow currently gives you nothing that would warn you.
The caveats#
Discovery is list-, marketplace-, topic-, and README-based rather than path-based, so the corpus over-represents repos that advertise their agent tooling. This is not a random sample of what sits in
.claude/across GitHub.Precision was validated; recall was not. The authors say plainly they do not know what the six gating rules miss, so the true defect rate could be higher.
Validation is model-on-model. Two LLM passes, no human scoring, and agreement between the adjudicator and the independent review was 70.7% (κ = 0.23) on contested pairs — weak agreement exactly where the calls were hardest.
Everything is public GitHub. Configurations maintained privately inside organizations may differ systematically: better in review discipline, plausibly worse in that no marketplace scan ever looks at them.
“Defect” here means an exposure decidable from bytes, not demonstrated exploitation. Nobody proved these setups got popped.
The takeaway#
What I'm filing away: the harness has become a distribution channel, and distribution channels get attacked. We went from .claude/ being a personal scratchpad to being something people install from a marketplace in about a year, without picking up any of the hygiene that package ecosystems spent two decades learning. Concretely, I'm doing two things after this paper — pinning MCP server versions the way I pin npm dependencies, and grepping every permission allowlist I maintain for interpreters and build runners (python, node, awk, sh, uv, make). It is a ten-minute audit, and this paper is a reasonable argument that roughly one in six of us needs it.
Paper: arxiv.org/abs/2609.07360 · Benjamin Kapner, Carmel Soceanu, Alicia Petrunin, Hofni Gartner · September 7, 2026