← all writing
05 · 14 Nov 2026 · 7 MIN READ

Don't ask your coding agent if it erred; ask where the spec should change

Most agent harnesses I've seen, including a few I've built, have some version of a "now review your work" step. Re-read the task, check the plan, look for mistakes. It feels responsible. A new paper from Purdue gives the clearest evidence I've seen that this step mostly re-approves whatever the agent already decided, and that a small change in who the agent thinks it's talking to fixes a good part of it.

The paper is Eliciting and Mitigating Specification Satisficing in Coding Agents by Ruixin Wang and Tianyi Zhang. The failure it names will be familiar to anyone who reads agent transcripts: the agent considers an implementation that meets a requirement, then switches to one that doesn't, and talks itself into it. Their example, from a cattrs task on DeepSWE, has the agent writing this in its reasoning before committing to a merge that can overwrite fields the spec says to preserve:

arguably not 'preserving', but likely acceptable

It then fails 2 of 69 tests. Nothing was misunderstood. The agent read the requirement, saw the right design, and picked the easier one anyway. The authors borrow Herbert Simon's word for it: satisficing, taking an option you've decided is good enough.

Self-review keeps signing off on the same shortcut

The core argument is simple, and I think correct. When you ask an agent whether it made a mistake, it judges with the same premises it used to make the choice. It already decided "preserving" could mean something looser, so on review the choice still looks fine. The paper's data backs this up. On DeepSWE (113 long-horizon tasks) with GPT-5.6-Luna on mini-swe-agent, four runs each:

DeCRIM is the one that should worry spec-driven folks. Breaking the task into a requirement checklist and asking the agent to tick each one is roughly what a lot of us do in our review prompts. Here it moved the needle by exactly zero.

The advisor framing does almost all the work

SatisFix intervenes once, right before the agent writes code, when most choices are made but nothing is built on them yet. It runs in two moves:

  1. Elicit. A message, framed as coming from the task's maintainers, asks the agent to act as an advisor and report where it concluded a requirement could be handled differently from what the task states: the requirement, its chosen implementation, and why. It explicitly does not ask the agent to judge whether that's a mistake.

  2. Reconsider. The next message says the maintainers won't change the task, puts the agent back in the developer role, and asks it to list alternatives for each reported choice and a plan to check them against the task and the repo, without settling anything yet. A final message tells it to carry out the plan, decide each choice, and finish.

In the cattrs example, the advisor-mode agent reports that its merge "violates a strict reading of 'preserve structured fields'" and calls it "a judgment call". On reconsideration it switches back to the right design and passes all 69 tests.

The ablation (single run, GPT-5.6-Luna) is the part I'd show a sceptical teammate. Full SatisFix hit 49.6%. Drop the advisor role and just ask directly whether the agent concluded a requirement could be handled differently: 43.4%, identical to the base agent. Replace both steps with a generic self-check at the same point, task description repeated: 44.2%. Keep elicitation but skip reconsideration: 47.8%.

So the same question, asked as "did you deviate?", gets nothing. Asked as "help the maintainers see where the spec would need to change", it surfaces the shortcuts. With GPT-5.6-Luna the advisor reports flagged satisficing in 61 of 113 tasks (90 items); with DeepSeek V4 Flash, 105 tasks (227 items). The most common reasons were reinterpreting the requirement and narrowing its scope ("that case won't come up").

Rereading the task isn't what helps. Giving the agent permission to admit a shortcut without calling it an error is.

Cheaper than Self-Refine, and it travels

The gain held across five backbones, two benchmarks and two frameworks. On DeepSWE, DeepSeek V4 Flash went from 53.32% to 57.08% and Qwen3.8-27B from 42.26% to 47.79%. On OpenHands with GPT-5.6-Luna it moved 46.90% to 51.33%. On the 51-task hard subset of SWE-bench Pro v2, every model improved, the biggest jump being Gemma 4 26B A4B from 9.80% to 21.57%. With DeepSeek it even cut token cost per task by 14%, because the agent took fewer steps (131.6 vs 152.9).

I'd hold these numbers a bit loosely. The DeepSWE standard deviations are around ±2.6 to ±3.0 points over four runs, so SatisFix vs Self-Refine (±3.98) overlaps more than the headline suggests. The full SWE-bench Pro set is mostly saturated for the stronger models (above 91% either way), so the hard subset of 51 tasks carries the argument there, and single-run results on 51 tasks move about 2 points per task. The ablation is one run. And the analysis of what the agents reported was labelled by Claude Code and Codex, with a human only checking disagreements, so errors both annotators share go uncaught. The authors say all of this themselves.

It backfires when the agent has cut too many corners

One result in the analysis cuts against the method. With DeepSeek V4 Flash, on tasks where the agent reported one or two satisficing items, SatisFix succeeded 67.6% of the time against 51.1% for the base agent. On the 36 tasks with three or more, it dropped to 44.4% versus 54.2%. The authors' guess is that once a plan rests on many shortcuts, reopening all of them at once does more harm than good. That matches what I see: a plan with five "good enough" decisions usually needs a rewrite, not five patches.

There's also the framing itself. The elicitation message claims to come from the maintainers, and a later message apologises and says it didn't. The paper doesn't measure whether that changes how the agent treats subsequent instructions. In a harness where the agent is also supposed to trust messages from real humans, I'd rather not teach it that "the maintainers" sometimes aren't.

The checkpoint I'm adding before the first edit

I don't need the fake-maintainer bit to steal the idea. In a Claude Code-style setup with a planning phase, the cheap version is a hook or sub-agent prompt that fires once, between plan and first edit, and asks the agent to write up, for the spec owner, every place its plan handles a requirement differently from how the spec literally reads, and why. No "is this wrong?". Then a second turn that says the spec is fixed, list alternatives, check them against the spec and the repo, and decide.

Two things I'll add on top. If the report lists three or more items, I'll send the task back to planning instead of patching each choice, given the DeepSeek result. And I'll keep the reports. A list of every place the agent decided my spec was "ambiguous" is a better spec review than most humans give me, and it tells me which sentences to rewrite before the next run.

My bet is that this framing trick ends up in harnesses before it ends up in training, simply because it costs about $0.25 a task and one extra prompt.


Working on something similar?

Say hello — I read every email.