On SWE-bench Pro, DeepSeek V4 Flash with no hints resolved 8.4% of 131 instances. Hand it just 1.2 identifiers per task on average, with their signatures, and it resolved 13.0%. Hand it the broadest hint set, 17.2 identifiers per task, and it resolved 22.1%. Which of those two numbers is the honest measure of the model?
That is the question in Between Spoilers and Guesswork, a paper from Rensselaer and IBM about hints in coding benchmarks. I read it as a paper about specs, because the thing they are measuring is the same thing I fight every time I write a task for an agent: how much do you tell it before you are doing its job for it?
Hints fix real ambiguity, but they can also hand over the answer#
Repository-level tasks are under-specified. The issue says "add a method that does X" and the hidden tests call it by a name nobody could guess. Benchmark builders patch this by appending hints. The authors split that into two choices: which identifiers to disclose (selection) and how much to say about each (projection).
Their selection rule, DefUse, keeps only identifiers that are defined in the gold patch and used by the test patch, or the reverse. Those are exactly the names an agent cannot infer but a test will demand. Their projection rule, Declaration, reveals name, kind, path, whether the file is new, and the signature. It deliberately leaves out any description of what the thing does.
The setup: four Python benchmarks plus Multi-SWE-bench Java, three models (DeepSeek V4 Flash, MiniMax M2.5, GPT 5.6 Luna), mini-SWE-agent harness, one trajectory per instance.
Structural hints are nearly free; prose hints are where the leakage lives#
The attribute ablation on SWE-bench Pro (DeepSeek, n=68) is the table I keep coming back to:
Empty: 10.3%
Name only: 22.1%
Declaration (name, kind, path, signature): 29.4%
Name plus semantic description: 42.6%
All hints: 72.1%
Oracle hint: 88.2%
Names alone more than double the pass rate. Everything past the declaration is prose, and prose is where the climb gets steep. The authors ran an LLM judge (GPT 5.6 Luna) over 787 identifier-attribute pairs and scored necessity, disclosure and guidance. Structural attributes scored necessity 1.56, disclosure 0.00, guidance 0.16. Semantic attributes scored necessity 1.50, disclosure 1.45, guidance 0.99.
Read that again: semantic descriptions are about as necessary as signatures, but they disclose nearly all of the solution. The judge's net margin is +1.39 for structure and -0.05 for semantics.
Higher solve rates alone cannot distinguish between resolving guesswork and supplying spoilers.
For benchmark design, that is the argument for DefUse plus Declaration. A score gain from hints should only count if the hint cannot do the thinking.
What I would change in a task spec this week#
I write a lot of specs that agents consume, and I habitually add a paragraph of "here is how it should work". This paper says that paragraph is the expensive part of the information budget, not the cheap one. What I would do instead:
List the names, file paths and signatures the tests or downstream callers will depend on. This is the part the agent truly cannot guess.
Keep behavioral prose for the genuinely ambiguous parts, and say so. If the description is a recipe, I am writing the patch myself.
When I evaluate my own agents on internal tasks, report results with and without a signature-only hint. The gap tells me how much of my failure rate is naming guesswork rather than reasoning.
The sizing result supports this. DefUse averages 1.2 identifiers per instance, versus 4.8 for Doc2Feat-bench, 7.7 for FEA-Bench and 17.2 for the broad Pro selection. A tiny, precisely chosen list already captures a good part of the gain, so a spec does not need to enumerate the whole API surface.
It also holds up across models in the projection comparison (DefUse, SWE-bench Pro, n=68). Declaration versus empty: DeepSeek 29.4% vs 10.3%, MiniMax 14.7% vs 2.9%, GPT 5.6 Luna 73.5% vs 55.9%. The weakest model gains relatively the most, but the strongest still gains 17.6 points. On Java (DeepSeek, n=29), empty hit 17.2%, Declaration 34.5% and the broad selection 48.3%.
How far I would trust the numbers#
There is one more reason I like the framing. It gives a failing agent run a diagnosis I can act on. If the model misses a name the tests demanded, that is a spec gap I can close with one line of signature. If it misses because the behavior was ambiguous, that is a real reasoning or intent problem, and a longer prompt will only hide it behind a leaked answer.
The samples are small. Several of these cells are a handful of instances, and the authors say plainly that they ran one trajectory per instance, used a single harness, and worked with subsets as small as 31 and 26 instances, so small differences are sensitive to stochastic variation. The leakage judge is also shakier than its headline suggests: agreement is 94% three-sample unanimity for signatures but only 41-42% for semantic descriptions, and the authors call the harm margin a heuristic, not a calibrated measure of information.
So I would not quote the -0.05. I would trust the direction: structure is cheap and non-leaky, prose is where the answer sneaks in. That matches what I see informally when a spec starts to read like pseudocode.
My bet is that spec-driven tooling will end up with a two-tier format: a machine-checkable interface section the agent must satisfy, and a short intent section kept deliberately lean. Whether the second tier can be measured for leakage the way this paper does it is the open question I would most like someone to answer.