Tests tell an agent what the program should do. They say almost nothing about how the program should be built, and that gap is where my specs keep leaking. Skeleton Coding, from an AWS team, puts numbers on it: a skeleton built from 20 observations beat 160 curated behavioral observations.
The paper's argument is simple. When you hand an agent end-to-end tests, the agent first picks a program structure, silently, and the tests then select among programs that structure allows. If the structure is wrong, more tests do not rescue you. The authors even prove a toy version of this, where one structure needs 2^d − 1 tests to pin down what another pins down with d.
no number of tests provided results in a program that passes all hidden tests
That quote is about their theoretical setup, and the authors note the guarantees do not carry over exactly to prompting. I read it as a framing, not a law. The empirical part is what matters.
What a skeleton is, and what it did to pass rates#
A skeleton is the program's structure written in code-like form: data types, modules, signatures, data flow, local contracts and global invariants, alongside the usual behavioral spec. They ran four models (GPT-5.6 Sol, GPT-5.6 Luna, Qwen3.8-27B, GLM 5.2) on three benchmarks: SlopCodeBench (35 problems), SDDBench (13 anonymized Python libraries) and ProgramBench (20 tasks with an oracle binary).
SlopCodeBench mean pass rate: 71.7% to 89.2%. Behavioral specs added at most 9.4 points.
SDDBench: 29% to 74.2%. Function stubs alone got 53% to 66.4%.
Across all models and benchmarks, relative test error fell 48.4% (40.7% to 21.0%) against the best end-to-end behavioral spec.
ProgramBench, where the oracle is already strong: 64.0% to 70.9%. Real, but much smaller.
Prose did not do the same job. Architecture documentation lifted SlopCodeBench from 61% to 71% and SDDBench from 40.1% to 52.8%, which is good, and still below code-like skeletons. So the format matters, not just the information.
The cheap model is where I would look first#
With global invariants, GPT-5.6 Luna went from 67% to 90% on SlopCodeBench and from 21.1% to 79.5% on SDDBench. It landed within 5.0 to 8.2 points of Sol at roughly 15× and 26× lower cost. The authors say skeletons narrowed the small-versus-large gap by about 52.4%.
If that holds in your stack, it changes routing. Instead of buying a bigger model to get past a plateau, you spend human or planner-model effort once on structure and let a small model fill in the bodies. That is a spec-driven development claim with a price tag attached.
Less erosion, which tests alone made worse#
Erosion fell from 0.66 to 0.56 on SlopCodeBench and from 0.41 to 0.24 on SDDBench with structural specs, up to 24.7% overall. Behavioral specs pushed it the other way, to as much as 0.73 and 0.43. More tests nudged agents toward patching to green, and structure pulled them toward code that stays organized. I find that more convincing than the pass-rate gains, because it is the failure mode I actually see when agents iterate on a growing codebase.
Where I would not trust it yet#
More spec is not monotonically better. The authors report that overly detailed contracts sometimes lowered accuracy, especially for smaller models. That is the same catch you hit when you over-constrain a CLAUDE.md, and it means the skeleton needs tuning, not maximizing.
The benchmarks also lean friendly. SDDBench projects are public, and anonymization does not fully rule out contamination. ProgramBench results depend on a reference binary, and the 160-versus-20 comparison uses curated observations, which most teams do not have. Acceptance is by executable tests, not proofs, so passing still does not mean the program is the intended one.
Two smaller results that matter for harness design#
First, iteration. In the repair setting, by round 8 the skeleton workflow reached similar error rates with 32% fewer reference-binary calls. If your agent loop pays for every check against an oracle, a staging environment or a slow integration suite, that is a direct cost cut, not just an accuracy bump.
Second, the ProgramBench result is a useful counterweight. Oracle access alone already gave 61.5%, and adding a skeleton with local contracts moved the mean from 64.0% to 70.9%, a 10.9% relative gain. Skeletons help most where the agent has to guess structure and least where a strong feedback signal already constrains it. That matches my experience: a rich test and execution loop hides a lot of spec sins, and a thin one exposes them.
The practical reading is that structure and feedback are partly substitutes. If you cannot give an agent a good oracle, give it a good skeleton. If you can give it both, expect the skeleton to matter less than the headline numbers suggest, and spend your effort on keeping contracts short.
What I would try this week#
Take one task where an agent keeps passing tests with ugly structure. Before the agent starts, write the types, module boundaries and function signatures as stubs with one-line contracts, and commit them. Have a sub-agent only fill bodies and forbid it from changing signatures. Compare against the same task with extra tests instead.
My bet: structure-first specs become a standard layer in spec-driven tooling within a year, and the open question is who writes the skeleton. If a planner model can draft it well enough that a human only edits, the economics in this paper get very good. If it needs a senior engineer, it stays a power-user trick.