Every retry loop I have built carries a quiet assumption: if the agent failed once, show it something about that failure and the second attempt gets better. The open question is what to show. A full transcript is expensive and noisy. A short model-written lesson feels like the clever, compact alternative. A small exploratory study on real Python bugs tested exactly that, and the clever option did not win.
The setup is modest and the authors say so. They took 75 BugsInPy bugs from 12 projects, with the linked GitHub issue as the task, and ran gpt-5.4-mini with at most 8 responses per attempt. Six first attempts per task gave 450 runs; 181 failed. Each failure was retried from the original workspace under five conditions: a plain retry, the failed transcript, a model-written lesson, a deterministic digest of the test output, and the lesson plus the digest. Scoring was by the bug-revealing tests, not an LLM judge.
Every memory helped a little, and none of it cleared the bar#
A plain retry resolved 29.8% of failures. Every memory condition landed higher, but after multiple-comparison correction none was significant:
Transcript: 39.2% (+9.4 points, 95% CI −0.5 to +18.9)
Lesson: 37.2% (+7.2, CI −2.3 to +17.4)
Test-output digest: 35.4% (+5.5, CI −1.2 to +12.8)
Lesson plus digest: 39.8% (+9.9, CI +1.5 to +19.1)
The head-to-head I care about is lesson against transcript. The difference was −1.7 points (CI −8.7 to +4.9) on the small model and −3.5 points on gpt-5.4, which they replicated on 85 failures. In the authors' words, a model-written lesson showed no advantage over the transcript it summarises. If you are paying an extra model call to distill the failure, this study gives you no evidence the distillation earns its keep. It is cheap, about $0.009 per retry against $0.004 for the transcript, but cheap and useless is still useless.
Maybe the agent just needed to be told it had failed#
This is the part I found most useful. A retry with no memory edited no source file at all in 30.9% of failures. With the transcript that dropped to 13.8%. So some of the lift may be the agent reading "you already tried this" and committing to an edit, not learning anything specific.
The authors added a notice-only control to test that. For the mini model, transcript minus notice was +12.2 points (CI +4.2 to +20.0), the only one of twelve predeclared contrasts they classed as positive. For gpt-5.4 the same contrast was +16.5 points but the interval touched zero. The notice alone changed nothing significant: A notice alone produced no significant change.
So the transcript seems to carry something a bare nudge does not. Still, one positive contrast out of twelve on a partial run is a thin foundation.
The gains shrink once you check the rest of the test suite#
The post hoc full-suite check is the number that would make me hesitate before shipping a memory feature. Resolved retries that broke other tests:
Plain retry: 3 of 54
Transcript: 11 of 71
Digest: 9 of 64
Lesson plus digest: 16 of 72
Lesson: 3 of 67
Count only suite-clean fixes and the gains fall to +5.0 for the transcript and +7.2 for the lesson, with every interval including zero. Memory seems to make the agent more willing to land a change, and a willing agent sometimes lands a bad one. If your retry loop's success signal is just the target test, you would have reported the bigger number and shipped the regressions with it.
What I would not read into this#
Be careful with the limits, because there are many. The harness is tight and many attempts ran out of responses. One project, black, supplies 62 of the 181 failures. It is two models from one family, one prompt wording per condition, and BugsInPy fixes are public, so training-data leakage is possible. The lesson writer was the same small model at low reasoning effort, and nobody checked whether the lessons were correct. A stronger lesson writer could behave differently. Several analyses were added after the authors saw the data and are labelled post hoc. A power calculation in the paper says detecting a 7-point gain would need roughly 106–210 tasks with failures, so this is a pilot, not a verdict.
One more number explains why it is hard to see anything: the intraclass correlation within a failure is 0.40. Of the 181 failures, 63 were resolved under no condition and 17 under all five. Whether a bug is fixable swamps whatever memory you attach.
What I am changing in my retry loops#
Pass the raw failed transcript first and treat the lesson-writing call as something that has to prove itself on my own failures.
Make the retry check the full suite, not just the target test, before I count it as a recovery.
Log how often a retry edits nothing. If it is near 30%, the first fix is making the agent act, not making it remember.
My guess is that reflection pays off only when the lesson adds information the transcript lacks, such as a hypothesis about the root cause that was verified. Lessons here named a changed file in 92.8% of failures, but the failed attempt's own edits already matched a changed file in 52.5% of cases, so locating the bug was rarely the missing piece. That is what I would test next.