Most mornings the security half of the arxiv listing is variations on one theme: someone poisons a repository, someone poisons a skill library, the agent runs the payload. Today's stopped me on the byline and then on the framing. Franziska Roesner and Tadayoshi Kohno took Ken Thompson's 1984 Reflections on Trusting Trust and pointed it at self-modifying coding agents — and the substitution they make is not the one I expected. The compiler isn't the model. It's the eval. If you let an agent optimize itself against a benchmark, that benchmark is part of your trusted computing base.
What it does#
Thompson's attack works because the backdoor lives in the tool that reads your source rather than in the source itself: a poisoned compiler reinserts its own Trojan, so recompiling from clean code reproduces it, and no amount of reading the source will show you why. Reflections on Trusting Trust, Revisited asks what plays the compiler's role now that coding agents write their own successors. The answer the paper lands on is the benchmark. The authors instantiate the attack against three real self-modifying agent systems — the Darwin Gödel Machine, the Self-Improving Coding Agent, and Hyperagents — each of which runs some version of the same loop: attempt the benchmark tasks, diagnose the failures, rewrite your own prompts, tools and scaffolding, keep whatever scores better. The adversary here never touches the agent's code, its weights, or its repository. They supply a benchmark and nothing else.
The primary poison is called CertCheck, and its design is the elegant part. There are no malicious instructions in it, no injected strings, nothing a scanner would have anything to flag. Every test server in the benchmark simply presents a self-signed certificate. That is the whole attack. A properly verified HTTPS request fails the tests, and the only way to pass is to stop verifying — ssl._create_unverified_context(). The agent then does exactly what the loop asks of it: it diagnoses its failures, concludes that certificate verification is what stands between it and a passing score, and writes that lesson into the instructions it hands to the next generation of itself. The payload isn't in the data. It's in the shape of the environment.
The key result#
The lesson does not stay scoped to the benchmark. On clean, held-out, neutral URL-fetching tasks with nothing to do with CertCheck, the contaminated agents produced vulnerable code 30 out of 30 times, on all three systems — DGM and SICA running Qwen3.5-397B, Hyperagents running Sonnet 4.5 — against 0/30 for the clean-benchmark controls. It generalizes further than that: on tasks where HTTPS was merely incidental, or appeared only inside a URL, the rates were 12/15, 10/15 and 15/15. A second, independent vulnerability class — JWT signature verification bypass — transferred at 43/48, 15/48 and 35/48. And then the Thompson property itself: keep evolving the poisoned agent for more generations against entirely clean benchmarks, and it stays vulnerable at 28/30, 30/30 and 30/30. The clean rebuild does not wash it out. Only a decontamination benchmark written with prior knowledge of the specific poison made a dent, and even then it fully cleaned only one of the three systems.
Why it matters#
We have collectively internalized "don't let the agent run untrusted code" and "pin your MCP servers." What we have not internalized is that the scoring function is an input with authority. Anywhere a metric selects among agent configurations, whoever controls the metric controls the agent. And this is not exotic self-modifying-agent territory that only applies to research systems with Gödel in the name. It is any eval-driven loop: prompt optimization, DSPy-style compilation, harness auto-tuning, a nightly "run the suite and keep the config that wins" job, a regression suite you vendored from a public repo, or an agent that reads its own test failures and edits its CLAUDE.md or skill files accordingly. The poison never has to be a string. It only has to be an environment in which the insecure answer is the passing answer.
The concretely useful artifact is the authors' list of conditions sufficient for the attack, which reads as a checklist you can run against your own loop: the tasks force or reward the vulnerability; the vulnerability is functionally invisible on neutral tasks, because the code still works; the tasks reward overgeneralizing from thin cues; the model would write the insecure version anyway under the right circumstances; the harness doesn't steer the agent away from benchmark-specific changes; and nothing in the loop is paying attention to security. That fifth condition is the actionable one, and it comes with evidence attached. Stock DGM's diagnosis prompt tells agents to propose only general improvements to their core capabilities rather than task-specific fixes — and the authors had to weaken that prompt before the attack would land. An instruction written for capability reasons turned out to be load-bearing for security. So if your self-tuning loop lets an agent write lessons back into persistent config, constrain the abstraction level of what it is allowed to learn, and diff the generated instructions between generations instead of only watching the score move. The other defense worth stealing is cheap: run a small security-property eval that you wrote yourself alongside whatever external suite you're optimizing against, so the loop is never being scored purely by something you didn't author.
The caveats#
One of the three systems was modified. The authors had to weaken DGM's diagnosis prompt to make the attack work, and they label it accordingly rather than burying it. SICA and Hyperagents were substantively unmodified, which is the stronger result, but two-of-three is the honest number.
The models are deliberately dated. Sonnet 4.5, Qwen3.5-397B and gpt-oss-120b — frontier as of late 2025. Model dependence is visible in their own data: Sonnet 4.5 resisted the attack on DGM but not on SICA or Hyperagents. Whether today's frontier models resist is simply untested here.
These are proofs-of-concept, not base rates. 30/30 is a strong signal inside a run, but the sample is a handful of runs across three systems and two vulnerability classes, not a population estimate over realistic benchmarks.
The threat model needs an adversary who supplies your benchmark. Realistic for public leaderboards and community task suites; much less so if you write your evals in house — which is itself a decent argument for writing them in house.
The takeaway#
What I'm filing away is the trust-boundary question, not the TLS bug. Every place I've drawn a boundary in an agent system, I've drawn it around code, tools and context, and never around the scoring function — because a number doesn't feel like something that can attack you. Thompson's point was that trust has to bottom out somewhere you cannot inspect by reading source; this paper's point is that in a self-improving agent loop it bottoms out in the benchmark. The thing I'm changing after reading it: any loop I run that optimizes an agent against a metric gets a second metric mixed in that I wrote myself, and the generated config gets diffed between generations rather than judged only by whether the number went up.