Tests can't see design. I have believed that for a while, mostly from watching agents turn in green suites with module layouts I would never accept in review. A new paper gives it a number: handing an agent recovered requirements and architecture docs moved held-out test pass rate by 2.1 points, not significantly, and moved module structure by 0.09 F1, very significantly.
The Code of Theseus, from Cezar Sas and colleagues at IDSIA/SUPSI and DFKI, is nominally a dataset paper. They recover stakeholder intent, requirements and a C4 architecture from 10,764 open-source repositories in Java, Python, C/C++ and C#. The interesting part is the test they built to check whether those documents contain anything real.
Rebuild the library, then compare the skeleton#
The recovery side is cheap by design. Instead of stuffing a repo into a context window, they feed the model a static dependency graph from Arcan plus a few context files (pom.xml, pyproject.toml, README, Dockerfile). That compresses inputs by roughly 91–96% in their examples; antlr4 goes from 653k tokens to 56k. Prompts are zero-shot, and each document is generated independently. Three open models wrote the documents: Gemma4-31B, Qwen3.5-122B and GPT-OSS-120B.
The check is a round trip, the "Ship of Theseus" in the title:
Take 29 Commit0 Python libraries, up to about 8k lines each.
Give a coding agent an interface contract (the names the tests import) and 30% of the tests it can run. The other 70% are held out for grading.
Run it with stakeholder intent only, or with intent plus the recovered requirements and architecture. Three runs per library, condition and generator, 1,000-step budget.
Compare held-out pass rate, and compare the module structure of the rebuild to the original.
Nobody needs a human-written reference design for this. The original code is the reference.
Pass rate shrugged, structure did not#
Held-out tests passed: 33.7% with intent only, 35.8% with the full documents. Delta +2.1 pp, CI [−1.6, +5.8], p = .144.
Module F1 on non-contract modules: 0.54 to 0.63, p = .002. Recall went from 0.51 to 0.63.
Win/tie/loss on module F1 across 24 libraries: 17/0/7. On correctness across 29: 19/1/9.
Agent steps barely changed (869 vs 858) while the task prompt grew from 1.0k to 4.8k tokens.
Read that again. The documents cost about 3.8k extra prompt tokens and did not make the agent run longer or pass meaningfully more tests, yet the agent built noticeably more of the architecture's modules (0.58 to 0.70 of them). The authors put it plainly:
artefacts "convey design that tests do not capture"
Their quote, not mine, but it is the sentence I would pin above any spec-driven workflow.
What this does to how I think about specs#
If you grade a spec-driven loop only on pass rate, you will conclude the spec did nothing. This paper says that conclusion is wrong, or at least incomplete. The spec changed what got built; the test suite just couldn't tell. Design intent is exactly the kind of thing that stays invisible until someone has to maintain the result.
The practical consequence for anyone building an eval harness around coding agents: add a structural metric next to the functional one. Module-level precision and recall against a known-good layout is cheap, and here it was far more sensitive than pass rate, with 29 libraries. In a real repo you do not have a reference layout, but you can diff against your own architecture doc or your last human-reviewed version.
A second, smaller thing I would steal: the compact representation. A dependency graph plus manifest files was enough to get coherent documents out of 120B-class open models. In their coherence check, requirements refined 63–72% of intent items and mapped onto 78–87% of architecture elements, against 5% and 10% for random pairings. For onboarding an agent to a legacy codebase, that beats dumping the repo.
Where I would not trust it yet#
The documents are good at sounding right. The LLM judge rated clarity and consistency high (Gemma's clarity 4.14, consistency 4.31 out of 5) but correctness only 3.02 for the best generator and 2.11 for the worst, and abstraction level was weaker still. Non-functional requirements were the lowest-scoring part (2.52 at best), which makes sense: a dependency graph does not reveal your latency or security targets. The judge is also uncalibrated and the authors flag possible same-vendor bias.
The experiment is narrow too: one agent, 29 small Python libraries, and the agent shares a model family with one of the generators. Pretraining memorization of popular libraries is a live worry; they mitigate it with a paired design and a sandbox without library copies, which helps but does not erase it. And with only 29 libraries, a real pass-rate gain of a couple of points would be undetectable, so "not significant" is not "zero". Their own exploratory checks also found no link between how much of the contract the docs covered and how much they helped, and longer documents did not help either.
Also notable: the cheaper, shorter Gemma documents did as well on structure as the longer Qwen ones. More words in the spec bought nothing.
What I am changing#
I will stop treating a flat pass rate as evidence that a design document is dead weight. Next time I compare an agent run with and without an architecture file in the repo, I will score the module layout as well, and I will keep the documents short. My prediction: structural metrics become a standard second axis in agent evals within a year, because pass rate keeps saturating while the code people actually have to read keeps diverging.