Skimming this morning's arxiv drop, one title made me stop mid-scroll: a placebo-controlled study of self-repair in code models. Self-repair — hand the model back its failing program plus the test output and ask for a correction — is such a load-bearing default in agent harnesses that I'd never seriously questioned whether the feedback is the part that helps. It's a single-author preprint run on consumer hardware, which almost made me scroll past; the control-group design is what pulled me back. The thesis I walked away with: most of what we credit to "self-repair" is just the second attempt, and showing a model its own broken code can actively make things worse.
What it does#
Try Again, Don't Look Back makes one sharp methodological point. The standard way to evaluate self-repair pits "retry with feedback" against "don't retry at all." That comparison quietly confounds two different things — the value of the feedback and the value of the extra attempt. If retry-with-feedback beats no-retry, we've been handing the feedback credit for a win that a blind reroll would also deliver.
So the author runs a placebo-controlled design on MBPP+ across three small Qwen2.5-Coder sizes (1.5B, 3B, 7B), with four matched-budget retry conditions: Resample (ignore the failure, sample the original prompt again), Placebo (show the failed program plus a content-free "this is wrong" notice), Feedback (show the failed program plus the actual failing input and expected-versus-observed values), and Reflect (feedback plus an instruction to diagnose the bug in prose first). Same token budget, same number of attempts — the only thing that varies is how hard the model is made to look back at its own mistake. That symmetry is the whole game, and it's what nearly every prior self-repair eval was missing. The effect then replicates on a second model family, DeepSeek-Coder.
The key result#
Blind resampling wins. On MBPP+ at k=8, the 1.5B model scores 0.749 resampling versus 0.688 with feedback — and 0.688 is exactly the placebo score, too. The informational content of the execution feedback added nothing measurable over a content-free failure notice; the entire gap is a 6.1-point penalty (p=0.006) you pay just for conditioning on the bad attempt. The pattern holds at 3B (0.778 vs 0.735) and only softens to a statistical tie at 7B, where resampling still gets there for 2.5–5.5× fewer tokens. The mechanism is anchoring: shown its own failed program, the model regenerates a near-identical one in 33–68% of retries, against just 2–14% under blind resampling — it commits to its first bad idea. Across six configurations spanning two families and two precisions, the size of the penalty is predicted by baseline quality alone (r=0.96): the better a model's first shot, the more it costs to make it stare at a rare failure.
Why it matters#
If you build agents, your retry loop almost certainly looks like the losing condition here. Run the tests, capture the traceback, paste it back with "the tests failed, please fix" — that is the Feedback condition, and on small models it's dominated by a strategy that simply throws the failed attempt away. The paper's real claim isn't "feedback is useless"; it's sharper and more useful: feedback is a costed resource, not a free default. You are spending tokens and inviting anchoring, and you should only reach for it when the feedback is rich enough to break the anchor rather than reinforce it. The author's prescription is blunt — retry rather than patch, cap the iteration budget near k≈2, and treat execution feedback as something you buy, not something you always attach.
Two follow-up experiments sharpen the lesson. Stuffing the context with other tasks' solutions (random or BM25-retrieved) changes nothing (±3.5 points), which localizes the harm to self-conditioning on your own failure rather than context length in general. And Reflect — making the model reason about the bug before rewriting — is the one condition that measurably loosens the anchor, yet it still doesn't pay for itself, losing to plain resampling at 2.5× the tokens. That's the part I'm sitting with: the intuitive fix, think harder about what went wrong, is real but not worth its price. It lines up with a thread I keep hitting in these papers — that the highest-leverage knob is usually the model's own generation quality, not the scaffolding we bolt around it. Concretely, it reframes a piece of harness plumbing everyone treats as obviously good — the test-output-in-the-loop retry — as a knob with a genuine downside, one worth A/B-testing against a dumb reroll before you ship it. Below 7B, that plumbing is net-negative.
The caveats#
Small models only. 1.5B–7B, quantized, run on personal hardware. At 7B the effect is already a tie, and a frontier model with a long, disciplined repair chain may genuinely use feedback. The anchoring mechanism plausibly persists at scale — but that's a hypothesis this paper doesn't test.
MBPP+ is short-horizon. Self-contained functions with cheap oracles. Whether anchoring still dominates in multi-file, long-horizon repo work — where feedback is far richer and a "reroll" costs a whole agent trajectory — is exactly where I'd want the follow-up.
Narrow spread. k≈2, one benchmark family; the elegant r=0.96 relation between baseline quality and anchoring cost is fit over a small range.
Single author, no funding. That doesn't make it wrong — the placebo design is more rigorous than plenty of well-resourced benchmark papers — but it's one person's worth of replication.
The takeaway#
What I'm filing away: the "does retry-with-feedback beat no-retry" comparison I'd half-consciously trusted is broken, because it never isolated the reroll. Before I add another self-repair turn to a harness, I want a placebo arm — does blind resampling at the same budget do just as well? If it does, the feedback is theater, and I'm paying tokens to anchor the model to its own mistake. What I'm doing differently after reading this: on cheaper models, my default retry becomes resample, capped near k≈2, and feedback becomes a deliberate, measured spend I have to justify — not the reflex it has quietly been.