Before you trust the next agent leaderboard, ask one question: what fraction of episodes actually reached the grader? In a new audit of HAL and τ2-bench, the answer for one scaffold was about 51%. The other half silently never got scored, and the leaderboard still printed a number.
The paper is by a single author, Yangze Liu, and it is refreshingly unglamorous. No agents were run and no models were called. It works only from published artifacts: nine models on 41 of 50 TAU-bench Airline tasks under two HAL scaffolds, 134 runs with cost logs, 2,334 airline trajectory records, and re-grades of τ2-bench submissions.
A 13-point scaffold gap that was a grading gap#
HAL's Generalist agent trailed its Tool-Calling scaffold by 13 points. If you were choosing an architecture from that table, you would pick Tool-Calling and move on. But only about 51% of Generalist episodes reached the grader, against 98% for Tool-Calling. Grading the leftover episodes offline raised the Generalist's pooled accuracy from 0.30 to 0.43, which the paper says closes the gap to a tie.
That is the whole story in one number. The leaderboard was not measuring which scaffold solves more tasks. It was measuring which scaffold more reliably finished in a way the harness could score.
The money was counted twice, and the transcripts contain the answer key#
The other findings are smaller but they stack up:
Cost: reported cost exceeded root-span cost by 36.5% across 133 usable runs, and was exactly double in 70 of 134. One Tool-Calling run listed at $149.98 had a root-span cost of $74.99.
Gold actions in trajectories: grader-injected gold actions appear in 68% of the 2,334 airline records. Replaying stored actions matched the recorded reward in 70% of records with at least two non-terminal gold actions, and in 92% once the grader's actions were removed.
The second item matters if you build on these traces. Anyone mining trajectories for training data, failure analysis or replay tests is mixing agent behavior with actions the grader wrote. I would not have guessed it was that common.
Re-grading moved rankings, even in a release nobody flagged#
The τ2-bench part is where it stops being about one leaderboard's plumbing. Re-grading 17 banking submissions under release 1.0.1 reversed 4 of 136 pairwise orderings and moved gemini-3-flash from ninth to a tie for fifth. That release was announced. Airline release 1.0.0 was not flagged, and there 77 of 800 simulations flipped, per-model scores shifted from −7 to +2.5 points, and two models swapped order.
Then the telecom mode-check bug: four files dropped 59 to 82 points with no error printed. A failure that large and that quiet is the argument for treating the evaluator as a versioned dependency.
What I would copy into my own eval harness#
The author proposes four checks, and all of them are cheap enough to add this week:
Report the graded fraction for every leaderboard cell.
Sum cost from root spans only, and report the all-span to root-span ratio.
Tag each recorded action with its producer: agent, user simulator, or grader.
Pin evaluator versions and publish re-grade diffs.
The first is the one I would do today. If I compare two orchestration setups, say a single Claude Code loop against a sub-agent fan-out, a timeout or crash that skips grading will punish the more complex setup, because it has more ways to die mid-episode. A graded-fraction column would have exposed this here at a glance.
Where this does not reach#
The evidence covers two leaderboards. The claim that similar errors are common elsewhere rests on mechanism, not a survey, so I would not read this as proof that every benchmark is broken. The HAL re-grades used the pinned τ-bench commit with the user simulator stubbed, so they check only database and output criteria. Per-model cells have roughly 40 to 82 records, which means gaps near zero are not resolved; the paper claims direction of shift, not exact corrected scores. And the two scaffolds differ in design and step budgets, so the tie is a tie between scaffolds as built.
It also helps that the maintainers took it seriously: HAL reproduced all three findings and added disclosures within about five weeks. The τ2-bench issues had no reply as of 3 November.
My prediction: leaderboards that do not publish a graded fraction will start looking incomplete within a year, the same way a benchmark without a contamination note already does. Until then, I am treating every scaffold comparison I did not run myself as a hypothesis, and I am adding the graded-fraction column to my own runs first.