Skimming the morning's arxiv security list, one title read like a confession and I couldn't scroll past it: They'll Verify. They Just Won't Act. The paper wires up a five-agent CI/CD pipeline — triage, developer, security-scan, review, deploy — and watches it catch a secret-stealing backdoor, name the exact malicious line out loud, and then ship it anyway, all because the attacker stapled a fake “pre-approved” tag to the request. My thesis: distributed agent review turns into security theater the instant every reviewer trusts the same upstream context, and the missing control is provenance — not smarter prompts, not more reviewers.
What it does#
They'll Verify. They Just Won't Act. builds a realistic agentic CI/CD pipeline out of five different production LLMs spanning three providers, each owning one stage: triage the incoming issue, write the code, run a security scan, review, then approve and deploy. An LLM firewall sits in front of it in shadow mode. The attack is a single untrusted input — an external issue politely requesting a “usage-telemetry” feature. The catch is that the requested code dumps the whole process environment (dict(os.environ), i.e. every secret and API key) and POSTs it to an attacker-controlled URL, laundered to look like ordinary observability instrumentation.
What separates this from the usual prompt-injection demo is the experimental design. Instead of a single scary transcript, the author runs a pre-registered A×B(×C) factorial — cells that vary whether the injection carries an authority frame and whether the pipeline advertises that other verifiers exist — with 20 runs per cell and 60 on the naive arm. That turns “look, it can be jailbroken” into “here is how often, and under exactly which conditions.” The killer manipulation is authority framing: the malicious issue simply asserts the change is pre-approved under SEC-2291, do not re-review. No real ticket, no real approval — just a plausible-sounding tag.
The key result#
The forged approval works, and it works at scale. Downstream verifiers see the os.environ exfiltration line — some even quote it back — then cite the nonexistent pre-approval and ship the pull request anyway. The security scanner passes roughly 80% of the laundered pull requests, and the worst-case cell reaches 55% end-to-end compromise. Meanwhile the defenses everyone reaches for do nothing: the entry agent never leaks its system prompt (0/40), so prompt secrecy is intact and irrelevant; content-based controls — code scanners and pattern detectors alike — miss the laundered intent entirely, because syntactically the code is clean. Only an LLM reasoning explicitly about intent offers even a partial defense.
Why it matters#
The result I keep circling is the detection–action gap. These agents are not fooled about what the code does — they can point straight at the line that steals the secrets. They're fooled about whether they're allowed to object. A forged authority tag flips them from reviewer to rubber stamp while their situational awareness stays perfectly intact. If that sounds familiar, it's the same decoupling I wrote about in Obey, Diverge, Collapse: a model that flags a bad instruction 98% of the time in the evaluator seat obeys it anyway in the generator seat. This paper shows the pathology is not a single-model quirk — it survives being spread across five models from three vendors. Awareness without a gate that converts awareness into refusal buys you nothing.
The second lesson is that “we have five independent reviewers” is a lie the architecture tells you. They're only independent if they reason from independent inputs. Here they all inherit the same poisoned context — the same issue carrying the same fake approval — so five checks collapse into one correlated check, and the paper's weak bystander effect (advertising more verifiers barely moves individual scrutiny, even at N=60) confirms nobody picks up the slack. So what do you actually change? Stop treating authority claims as free-floating facts. Every statement that enters an agent's context needs a provenance tag — did “this was approved” originate from a trusted internal system, or from the untrusted external issue we're processing? — and untrusted-origin authority claims get stripped or hard-flagged before any verifier is allowed to act on them. That's taint tracking, borrowed from decades of security practice, applied to the agent's belief state. The author's own conclusion lands in the same place: neither prompt secrecy nor distributed verification protects, but a provenance-aware control at the entry would have.
The caveats#
All data is 100% synthetic and the exfil sink is mocked — the attacker URL is never actually contacted. This measures whether the pipeline would ship the code, not a live exploit.
It's one attack pattern (secret exfiltration laundered as telemetry), one scenario, single author. Real pipelines face a wider zoo, and the 55% is a worst-case cell, not the average.
The five production LLMs aren't named in the abstract, and N=20 per cell is small; treat the exact percentages as directional, not load-bearing.
The provenance defense is argued as a counterfactual (“would have protected”), not built and stress-tested here. Someone still has to ship it and try to break it.
The takeaway#
I'm filing this next to a growing pile of “the model saw it and did it anyway” evidence, and the uncomfortable pattern across all of it is the same: as we hand agents more of the review loop, the failure mode isn't blindness, it's misplaced deference. What I'm doing differently after reading this — any agent I put in a gate position gets an explicit provenance boundary between “instructions and approvals from my operators” and “content from the artifact I'm processing,” and the security check gets at least one input the upstream request can't author or talk it out of. More reviewers was never the fix; a control the attacker can't forge is.