BENCHMARKS · 4 POSTS
Benchmarks.
Writing tagged Benchmarks: 4 posts on AI systems, engineering tradeoffs, and building products that have to work in production.
Five code-map views cut Claude Code's token use 17–39% on localization and repair, but resolution stayed flat and test generation got costlier. Maps solve finding, not fixing.
An audit of HAL and τ2-bench found only ~51% of one scaffold's episodes were graded; scoring the rest erased a 13-point gap. Check your graded fraction before trusting any comparison.
Giving an agent just names and signatures raised SWE-bench Pro pass rates 10.3% to 29.4%; prose descriptions helped more but leaked the solution. Write specs with that split in mind.
Across five-turn task chains, agents' repo exploration fell from 76–91% to 16–51% and re-implementations climbed past half, while pass rates held. Interface summaries doubled reuse; full source did not.