EVALS · 26 POSTS
Evals.
Writing tagged Evals: 26 posts on AI systems, engineering tradeoffs, and building products that have to work in production.
A placebo-controlled study of 5,835 coding-agent runs found telling the agent its token budget never beat an unrelated sentence. The cap itself is where the leverage is.
A meta-analysis of 516 multi-agent setups finds they tie or trail compute-matched single agents 74% of the time. Coding agents can escape that, but only through tests and split context.
Stale AGENTS.md lines barely changed task success across 732 runs, but a dead test command produced silent no-op checks in 64 of 80 runs and no agent ever fixed the file.
Trajectory-based failure predictors for coding agents added nothing over issue difficulty plus model identity, and model switching gains were within noise. Test your alarms against run-blind baselines.
In Multi-step SWE, only 12% of failed steps used their full time, and 52% ended with the agent claiming completion. Treat "done" as a trigger for independent checks, not a verdict.
An audit of HAL and τ2-bench found only ~51% of one scaffold's episodes were graded; scoring the rest erased a 13-point gap. Check your graded fraction before trusting any comparison.
On 181 failed bug-fix attempts, a model-written lesson matched the raw transcript (−1.7 points), and gains shrank once the full test suite was checked. Skip the distillation call until you measure it.
A harness cut 3,827 of 11,189 tool results into broken JSON while using 5.75% of the context window. If your agent seems oddly blind, audit truncation before blaming the model.
Across 147,002 coding-agent traces, early-failure predictors were at chance within a task (AUC 0.50), and plain continuation beat restart on solve rate and cost. Don't abort on prefix signals.
Across 3,000 turns, five models in Codex and Claude Code churned up to 31x the original code, and often kept editing after declaring the task done. Cap loops by diff budget, not the agent's verdict.