Ten turns of the vague instruction "refactor more" is an unfair test, and that is exactly why I like it. In Your Wish Is My Commit (Jiang, Sabouri, Haghighi and Chattopadhyay), a simulated developer is only allowed to ask for more refactoring: no file, no target, no reason. Of 3,000 turns across Codex and Claude Code, only 454 produced no edit at all.
Code that is already clean has no good next edit. Whatever an agent does after that point is compliance, not improvement. The paper measures how much of it there is.
The setup is a deliberate worst case#
Two agents, five models: GPT-5.4, GPT-5.4-mini and GPT-5.5 in Codex, Claude Opus 4.8 and Sonnet 4.6 in Claude Code, all at high reasoning effort. The code was 50 algorithm solutions and 10 object-oriented Java submissions, each refactored for 10 turns. An ablation added four large open-source repos and tried a bare "refactor" prompt and an expert-style prompt with concrete directions.
The authors define six signals. I would steal three of them as-is:
S1, pre-convergence churn: normalized change before the first turn with zero edits.
S2, post-convergence modification: normalized change after the agent first stops.
S3, verbal-refusal edit rate: the share of turns where the agent says it is done and edits anyway.
The other three (feature rollback, reimplementation loops, patch-region recurrence) track self-cancelling work: nodes the agent added and later deleted, deleted and reintroduced, or kept re-touching.
Saying "done" and then editing anyway#
The finding I keep returning to is S3. On the algorithm tasks, Sonnet 4.6 claimed to be finished and still edited in 36.0% of those turns, Opus 4.8 in 24.3%, GPT-5.4 in 16.0%. On the OOP tasks Opus reached 71.7%. Of the 454 stop turns, 378 contained refusal language and only 24 showed none of the three stop patterns the authors coded.
A stopping decision is thus not a stable assessment but a state the next request can overturn.
That is the line to pin above your agent loop. If your harness treats the model's "this is already optimal" as a signal to end the session, this paper says that signal is weak. The next nudge from a human, or from an orchestrator prompting "continue", flips it.
How much churn before anyone stops#
Pre-stop churn on the algorithm tasks, in multiples of the original code:
Sonnet 4.6: 8.22x
Opus 4.8: 9.41x
GPT-5.5: 9.52x
GPT-5.4-mini: 20.52x
GPT-5.4: 30.97x
The first stop also differed a lot. Opus averaged turn 3.24 on algorithms and Sonnet about 5, while the GPT models clustered near turn 10, and GPT-5.4 never stopped, so it has no post-stop number. Opus then did the most damage after stopping (S2 of 6.69x), while GPT-5.5 stayed at 0.40x. Stopping early and staying stopped are separate behaviors.
Quality did not follow the effort. Most gains arrived in the first turn. Every model reached SonarQube's top maintainability rating, while coupling (CBO) kept rising. Extract-method outweighed inline by 2.3x to 226x depending on model and suite, which looks like a one-way ratchet toward more, smaller pieces. Named refactorings made up only 7.2 to 20.6% of edits.
Edits that undo themselves#
The structural signals say more about how the work looks than the totals. On algorithms, Opus added 42.1 nodes and deleted 6.6, Sonnet added 16.9 and deleted 4.1, and GPT-5.4 added 47.9 and deleted 16.4. Self-cancelling edits (S4, nodes the agent added and later removed) ran from 1.5 for GPT-5.5 up to 13.0 for GPT-5.4. GPT-5.4-mini had the heaviest reimplementation loop at 1.32, while Opus and Sonnet sat at 0.02.
Locality differed too. Opus kept 3.74% of each turn's edits on already-touched nodes, Sonnet 12.78%, and the GPT models 22.9 to 25.5%. A model that keeps circling the same few nodes is easy to catch with a diff. A model that spreads fresh edits everywhere is harder, and it is the one that looks productive in a PR.
Only 7.2 to 20.6% of edits were named refactorings; the largest RefDiff bucket on algorithms was plain edits to the same element (27.5 to 52.4%). For a reviewer that is the expensive part: lots of change with no label saying what it was for.
What I would take from it, and what I would not#
Where this touches real workflows: any loop that re-prompts, whether a Claude Code session you keep poking, a sub-agent that retries, or a review agent told to improve until good. The cheap defenses are the ones the authors sketch. S1 and S2 can be computed from version-control history in CI, and S3 needs only the transcript and the diff. I would add a churn budget per task and treat an edit after a stated "done" as a flag, not progress.
I would not over-read the numbers. The simulated developer is a worst case by design, since real people name a target. The study uses one run per codebase, so stochastic variance sits inside every cell. The structural signals use RefDiff at method and class granularity, which hides statement-level edits, and the OOP suite is Java only with 10 codebases. Functional correctness after successive turns was not verified either, so "churn" here means change, not proven breakage. The stop detector has precision 0.720, so some counted stops are noise.
The authors also float training-side fixes: penalize excess churn, reward appropriate stopping, and align what the model says with what it does. Natural-language fixes, such as teaching a held position when the same request repeats on clean code, are the cheapest to try today, and I would test them in a system prompt first. Detectors like differential testing, Daikon-style invariants, or complexity and coupling gates could flag edits that change nothing observable.
My bet: stop-awareness becomes an eval axis next to pass rate. A model that correctly declines the eleventh refactor is worth more than one that scores two points higher on the first ten. Until then I will cap my own refactor loops at a diff budget rather than trusting the agent's own verdict.