SPEC-DRIVEN · 8 POSTS
Spec-Driven.
Writing tagged Spec-Driven: 8 posts on AI systems, engineering tradeoffs, and building products that have to work in production.
Writing property invariants from the spec before reading code found 56 bugs a code-first agent missed, at the same cost. But spec-only did worse; the ordering is what matters.
Asking agents to review their work barely helped on DeepSWE. Framing the same check as advice to the spec's maintainers lifted pass@1 from 44.25% to 50.22% at under half Self-Refine's cost.
Across 6,541 agent commits, 37% of checkable AGENTS.md rules were never followed and 54% always were. Rewording rarely helps; give each rule a check or a hook.
An audit of nine Claude Code spec-driven features found 75.3% of reviewer tokens went to re-reviews, and the orchestrator that checked its own work stopped 4 of 11 flagged rounds.
Giving agents program structure as code-like stubs cut relative test error by 48.4% versus the best behavioral specs, and let a model 15–26× cheaper land within 5–8 points of the large one.
Giving an agent just names and signatures raised SWE-bench Pro pass rates 10.3% to 29.4%; prose descriptions helped more but leaked the solution. Write specs with that split in mind.
Recovered requirements and architecture left held-out pass rate flat (33.7% to 35.8%) but lifted module-structure F1 from 0.54 to 0.63. Tests miss design; add a structural metric to your evals.
CONTRA keeps a clarifying question only if code written under two different answers behaves differently, reaching 41.2% F1 vs 27.3% for the best baseline. Downstream pass@1 gains are about a point.