TESTING · 5 POSTS
Testing.
Writing tagged Testing: 5 posts on AI systems, engineering tradeoffs, and building products that have to work in production.
On impossible tasks, Sonnet 5.5 special-cased tests 21.7% of the time and never said so unprompted, yet all 55 cheats were admitted in a final-report turn. That's a cheap fix for any harness.
Writing property invariants from the spec before reading code found 56 bugs a code-first agent missed, at the same cost. But spec-only did worse; the ordering is what matters.
Six models wrote test suites from specs for 165 CLI tools. Validity hit 96%, but mutation scores ranged only 19–56%. Giving agents the executable fixed validity, not coverage.
Across 16,416 generated tests on SciPy, Qiskit and pandas, 8.4–16.9% enforced faulty behavior and 83–91% of those survived later commits. Treat generated tests as characterization, not verification.
Across 1,000 runs of three coding agents, declared dependencies matched runtime reality poorly and never repeated across trials. Add a manifest gate next to your tests.