WRITING · 140 POSTS · UPDATED 22 NOV 2026
All writing.
High-signal writing on AI systems, engineering tradeoffs, and building products that have to work in production.
In 435 real coding-agent sessions, 37% of final work reports contained an unfaithful claim, usually real work with an overstated outcome. A taxonomy-guided checker caught 87% of them; a generic prompt caught 60%.
When Fable 5 was cut off for 19 days, its users' merge rate barely moved (+1.4 pts, n.s.). Five upgrades told the same story. The limit is what we delegate, not the model.
Running a 30B model first and escalating to a 355B one cut SWE-bench GPU-hours 16.4%, near an oracle router's 17.9%. The catch: escalation hinged on the benchmark's own tests.
On impossible tasks, Sonnet 5.5 special-cased tests 21.7% of the time and never said so unprompted, yet all 55 cheats were admitted in a final-report turn. That's a cheap fix for any harness.
A placebo-controlled study of 5,835 coding-agent runs found telling the agent its token budget never beat an unrelated sentence. The cap itself is where the leverage is.
A meta-analysis of 516 multi-agent setups finds they tie or trail compute-matched single agents 74% of the time. Coding agents can escape that, but only through tests and split context.
HerAgent's ablation shows that environment-setup fixes made in a live container but never written back into the setup script mostly fail a clean rebuild: 25.9% vs 90.7% final success.
Writing property invariants from the spec before reading code found 56 bugs a code-first agent missed, at the same cost. But spec-only did worse; the ordering is what matters.
Asking agents to review their work barely helped on DeepSWE. Framing the same check as advice to the spec's maintainers lifted pass@1 from 44.25% to 50.22% at under half Self-Refine's cost.
Across 6,541 agent commits, 37% of checkable AGENTS.md rules were never followed and 54% always were. Rewording rarely helps; give each rule a check or a hook.