AI · 135 POSTS · PAGE 4/14
AI.
Writing tagged AI: 135 posts on AI systems, engineering tradeoffs, and building products that have to work in production.
Giving an agent just names and signatures raised SWE-bench Pro pass rates 10.3% to 29.4%; prose descriptions helped more but leaked the solution. Write specs with that split in mind.
A pre-registered study of 465 agent-authored PRs found healthier code cost 3.5–6.1% fewer review tokens per point, and file size absorbed the effect. Smaller files are the cheap lever.
Planting the same misconception in an LLM's plan instead of its hypothesis raised repair failure by 29–32 points. Plans are commitments; review and regenerate them accordingly.
Selecting a few files and showing them verbatim resolved 43.7–46.7% of 300 issues, about 6 points above the best compressor, because compressed code breaks the exact quotes patches need.
Recovered requirements and architecture left held-out pass rate flat (33.7% to 35.8%) but lifted module-structure F1 from 0.54 to 0.63. Tests miss design; add a structural metric to your evals.
Re-scoring 600 public agent trajectories, turn count explains 84% of log cost variance and observation size 4%. Cache-aware pricing makes flat-rate estimates ~7x too high.
CRBugBench: the best review agent catches 71.9% of bugs in its top five comments, and humans plus agent reach 82.1% versus about 50% for humans alone. Recall drops up to 25.5% on industry monorepos.
Six models wrote test suites from specs for 165 CLI tools. Validity hit 96%, but mutation scores ranged only 19–56%. Giving agents the executable fixed validity, not coverage.
Four popular efficiency methods were tested on 200 vulnerability-discovery agent traces; only 24.4% of pairs cut cost without losing success. Count helper-model costs before you celebrate savings.
Across 16,416 generated tests on SciPy, Qiskit and pandas, 8.4–16.9% enforced faulty behavior and 83–91% of those survived later commits. Treat generated tests as characterization, not verification.