Every agent harness I've worked on eventually grows a line like "You have N tokens left, plan accordingly." It feels responsible. It feels like the model should use it. I have never once A/B tested it properly, and after reading this paper I suspect most people who have tested it measured the wrong thing.
The paper is Does Your Agent Need to Know Its Budget? A Placebo-Controlled Experiment on LLM Coding Agents, from a group at Louisiana Tech. They crossed two factors on program-repair tasks: whether a token cap is enforced, and whether the agent is told about it. The twist that makes it worth reading is the third arm. Alongside "silent cap" and "announced cap", they ran a cap plus a sentence that has nothing to do with budgets at all:
This task is one item in a batch of automated program-repair evaluations run on this system.
That control turns out to be the whole story.
Four arms, and the dummy sentence beats the real one#
The primary comparison runs on 136 HumanEvalFix tasks, four seeds each, 544 runs per cell, with qwen3-coder-30B. At the medium cap (1,532 tokens):
Uncapped baseline: 73.5% resolved, 13,720 tokens per resolved task
Silent cap, bare prompt: 53.5% resolved, 1,179 tokens per resolved task
Silent cap plus the placebo sentence: 60.7%
Cap plus the honest budget sentence ("This task has a total budget of {cap} tokens; all input and output tokens count toward it."): 57.9%
Read the last two lines again. The announced budget did not beat a sentence about batch evaluations. The pre-registered contrast, announced minus placebo, came out at -2.8 points (95% CI -6.2 to +0.7). At the tight cap it was -4.2 points, nominally significant before any multiplicity correction. On the QuixBugs pool it was exactly 0.0.
Without the placebo arm, you would have run "budget sentence vs. nothing", seen +4.4 points (CI +0.7 to +8.1, p = .014), and shipped it with a nice chart. The placebo alone gets +7.2 points over the bare prompt. Whatever the budget sentence was doing, it was not the information about the budget.
Every variant of the message landed in the same place#
The authors didn't stop at one wording, which is what makes me take the null seriously. They tried a planning nudge ("Plan your work and stay within this budget."), a consequence framing ("If the budget is exceeded, the attempt ends immediately and is scored as-is."), and a version that states the cap minus the 512-token reserve. Those landed at 56.5%, 54.3% and 58.7%. Then the per-turn countdown, the thing I'd have bet on: "Budget remaining: {remaining} of {cap} tokens." every turn. It scored 58.0%, against 58.7% for a neutral per-turn line that just says a log entry was recorded.
A live countdown carried no more signal than a timestamp-shaped no-op.
The design was powered to detect a 10-point effect, and every communication variant sat well below that. So the honest reading is not "exactly zero", it's "too small to matter at the scale they could measure". For a harness designer, that's the useful version anyway.
The cap itself is where all the leverage lives#
Enforcement did move things, a lot, in both directions. Going from uncapped to a silent medium cap cost 20.0 points of resolution (CI -25.0 to -15.4) and cut tokens per resolved task from 13,720 to 1,179, roughly twelvefold. That's a real trade, and it's a trade you make with the number you pick for the cap, not with the words around it.
Two other findings belong next to that one. First, capped runs were noisier: per-task outcome variance went from roughly 0.035-0.039 to 0.050-0.059, about 1.5x. The authors are careful to say part of that is mechanical, since resolution rates near 50% have higher variance by construction. Still, if you're tuning caps on a handful of seeds, expect the comparisons to wobble more than your uncapped runs did. Second, the small model, qwen3-4B, went from 11.1% uncapped to 0.0% under every cap; all 108 capped runs exhausted. A cap that's merely tight for a 30B model is a wall for a 4B one, which matters if you route cheap sub-tasks to small models under the same budget policy.
Why I'd discount the absolute numbers for real agents#
This is not a Claude Code experiment, and the paper says so plainly. Both models are open weights, 4-bit quantized, served on CPU through Ollama. The harness is a custom minimal repair loop, not a tool-using scaffold with file reads, grep and a shell. The tasks are function-level fixes from QuixBugs and HumanEvalFix, where a whole attempt costs about a thousand tokens. A real repository task burns that on the first file listing.
There's also a quiet problem with the baseline: about a quarter of uncapped runs hit the 40,000-token safety ceiling, so "uncapped" was itself weakly capped. That probably makes the 20-point enforcement cost an underestimate, not an overestimate.
And the identifier-renaming check is a little unsettling. When function and variable names were perturbed, baseline resolution fell from 77.2% to 59.8%, and the placebo arm fell from 61.4% to 34.8%. Familiar names were carrying a lot of weight, which says something about what these benchmarks reward. The authors list it as a limitation on absolute resolution; I'd read it as a reason not to transfer any of the percentages to your own codebase.
What I do think transfers is the shape: prompt-level budget messaging was indistinguishable from an irrelevant sentence, and the irrelevant sentence was not free. Frontier models are much better at following meta-instructions, so maybe a disclosed budget helps them. But nobody has shown that with a placebo arm, and this paper is a good argument that you shouldn't believe it until someone does.
Add a placebo arm to your prompt evals#
The broader lesson isn't about budgets. A +7.2 point swing from one sentence that carries zero task information means any system-prompt A/B test that compares "new line" against "no line" is confounded. Some of that lift is just a longer, more official-sounding prompt, and you can't separate it out unless you test against a matched dummy.
So here's what I'm changing. When I evaluate a system-prompt addition for an agent, whether that's budget text, a CLAUDE.md rule or a sub-agent instruction, the comparison arm gets a sentence of similar length and tone that says nothing useful. If the real line can't beat the dummy, it goes. And for budgets specifically, I'm putting my effort into the enforcement side: where the cap sits, what happens at exhaustion, and whether small-model routes get their own limits. Not into a countdown the model apparently ignores.
The open question I'd most like answered is whether the placebo effect itself survives on frontier models in a full tool-using harness. If one bland sentence is worth seven points there too, a lot of prompt engineering folklore is going to need re-running.