The gate returned 0.250 in all 40 runs. Zero spread, no error, no warning. It was an LLM judge whose only job was to score how diverse a training set was, and in the deployed setting it had quietly stopped looking at the training set. That detail sits in an appendix of a paper on unattended agent self-improvement, and it is the part I would read first.
The paper, by De Wang and Peng Qi, describes a platform that reads production traces, classifies failures, and maps each one to a change: a system prompt edit, a tool definition, a skill, or fine-tuning a small model. A verifier approves or rejects the candidate, and the change ships with no human author. Their question is how much of that loop can safely run alone.
A judge that looks healthy while approving anything#
The diversity gate scored training-set diversity from 0 to 1. The authors varied judge model, prompt wording, imbalance level and breadth: 172 settings, 10 runs each. In 9 of the 37 configurations they examined closely, the score was identical at all four imbalance levels. All nine used the production judge, claude-haiku-4-5.
They are careful about this. That judge ran 33 of the 37 configurations, so the raw count is not significant (Fisher p=0.55). The cleaner evidence is the matched comparison: on identical inputs, the production judge stayed flat while claude-sonnet-5 and claude-opus-5 tracked imbalance. Scores did vary from run to run in 98 of 172 settings, so this is not just deterministic decoding.
The fix was almost embarrassingly cheap. Rewriting the criterion as a threshold ("if the largest seed exceeds 50%, score at most 0.10") held in 30 of 30 runs above 50% imbalance. The original wording held in 0 of 30. A vague rubric turned into a rule the model could not ignore.
Two tests I am stealing today#
The appendix gives two checks for verifier health, and neither needs a reference verifier:
Delete the signal. Remove the property the gate should judge entirely, not set it to neutral, and run both versions the same number of times. Identical scores mean the gate ignores the property. Their run took 40 calls.
Check the guarantees. List the request settings the gate relies on (forced tool call, fixed output format, temperature) and send an input that would be answered differently if the setting were ignored.
The second test caught three real failures in their stack. Forced tool calls cannot be combined with extended reasoning, so the API rejects that request. A gateway forwarded fixed-format requests without the format, silently. The judge models reject temperature settings. These were measured on one provider and gateway on 2026-08-13, so treat them as an example of the failure class, not a current fact about your stack.
If you run an LLM judge as a merge gate or an eval grader, this is the cheapest audit you can do this week.
Rollback only counts if you kept the old version#
The second precondition is recoverability: can you undo a bad change, and does the undo itself need a verifier? Their experiments compare two kinds of undo on τ²-bench airline tasks.
Restoring a saved copy brought behaviour back to the original within noise (difference 0.000, equivalent within ±0.05).
Asking for a regenerated undo did not. Three undos reached agreement with the original of 0.75, 0.83 and 0.90, against a noise floor of 0.98. The gate refused all three.
A "revert" the model writes is a new change, not a revert. That sounds obvious until your agent harness auto-fixes its own regressions by prompting itself to undo the last edit.
One task in six changed, and the average hid it#
The number that stuck with me: four admitted changes were deployed one at a time on 50 airline probe tasks (one run was lost, so three are reported). Outcomes shifted on 24%, 28% and 24% of tasks against a null-change floor of 8%. That is an excess of about 17 points, roughly one task in six, and each change was individually significant.
Flips went 18 up and 20 down. Mean pass rate barely moved. If you only watch the aggregate, a deployed prompt edit looks like nothing while it reshuffles which tasks pass. And per the authors, no undo repaired the effect of the time the change was live.
Even a decent judge is a wide estimate#
On 80 airline test trajectories, the authors scored verifiers against a trusted reference (database state plus communication checks). The LLM judge with natural-language assertions reached a strength of 0.70, but the confidence interval ran from 0.36 to 1.00. It admitted 10% of bad trajectories and rejected 21% of good ones. Plain database equality matched the reference exactly, and a verifier that admits everything passed all 42 failing trajectories. The lesson I take: where a deterministic check exists, use it, and give the judge only what the check cannot see.
Where I would not lean on it yet#
The authors say the rule comes from one platform and was applied after the fact. Only prompt edits were tested end to end, and the benchmark was τ²-bench airline plus retail, not production. The verifier-strength experiment was underpowered: the slope of regret against strength was not significant (p=0.23), and they estimate about 220 held-out tasks are needed versus the 20 they used. So the claim that weaker verifiers cap improvement is argued, not demonstrated here. The measured, solid parts are the collapse, the rollback gap and the blast radius.
My bet: the useful idea is not the thresholds but the refusal state. A pipeline with admit, reject and refuse, where refuse means a precondition was never measured, is a better default than one that treats an unmeasured verifier as a passing one. I am adding a delete-the-signal test to every judge I rely on before I trust another score from it.