Plan mode feels like a safety feature. You make the agent write down what it intends to do, you read it, and only then does code get written. A paper I read today made me less comfortable with that story: the plan is also the place where a wrong idea does the most damage.
In Poison Valleys: Diagnosing Path-Dependent Failure in LLM Code Repair, Veljko Skarich plants a plausible but wrong idea at a chosen stage of a repair and watches what happens. The idea goes into the model's hypothesis (what it thinks is wrong), its plan (what it will do about it), or both. Same misconception, different commitment point.
Same bad idea, 30 points more failure once it sits in the plan#
The setup is small and controlled: two synthetic Python repair tasks on real packages, a SQLite-backed queue that needs an atomic batch-claim, and a NetworkX multi-source shortest-path task that needs source-dependent edge filtering. There are 11 and 8 families of misconception respectively, five models (including Claude Haiku 4.5 and Claude Sonnet 5), and a binary oracle for pass/fail. Of 1,844 requests, 1,340 made it all the way through the oracle.
Raw failure rates when the poison sits in only one stage:
PersistQueue: 21.9% failed with the idea in the hypothesis, 54.7% in the plan.
Dijkstra: 53.4% in the hypothesis, 83.0% in the plan.
After balancing the experimental mix, moving the misconception from hypothesis to plan raised failure by 32.2 points on PersistQueue and 28.9 points on Dijkstra. The direction survived every leave-one-family-out and leave-one-model-out check the author ran.
The paper's explanation is the part I keep thinking about:
A plan commits the repair to operations, owners of state, update order, error handling, and compatibility choices.
A wrong belief is something the model can still route around. A wrong plan is a list of edits, and the model is very good at carrying out lists of edits.
Writing the plan down made it a commitment, not a draft#
This is where the spec-driven angle gets uncomfortable. We push agents toward explicit plans, task lists and specs because they make work reviewable. The data says the same artifact makes errors stickier. Once a bad idea has been turned into steps, the implementation inherits its structure: who owns the state, in what order it updates, how errors are handled.
The author is explicit that this is not an argument against planning: "This does not mean that planning is harmful." I agree. The practical reading is that a plan deserves more scrutiny than the diagnosis in front of it, because the plan is where the model stops being able to reconsider cheaply.
What I would change in a plan-then-act loop#
Review plans for mechanism, not intent. "Add an atomic claim" reads fine; how state is owned and in what order it updates is where the poison lives.
Let the agent challenge its own plan against the failing test before it edits anything, instead of treating plan approval as the last gate.
Keep the hypothesis and the plan as separate artifacts. If the diagnosis turns out wrong, regenerate the plan from scratch rather than patching it.
Adding a matching hypothesis to an already poisoned plan barely mattered: +6.3 points on PersistQueue (CI 0.5 to 12.2) and +2.2 on Dijkstra (CI spanning zero). Once the plan is wrong, the diagnosis adds little. That fits the idea of regenerating the plan instead of editing it.
Models disagree about how much a bad plan costs#
The per-model spread is the most useful thing in the appendix for anyone choosing a model for a planning step. On Dijkstra, one intervention (a plan-only poison called B01) left GPT-5.6 Luna passing 119 of 340 candidates (35.0%) and Qwen 3.8 Flash passing 6 of 127 (4.7%). Same poisoned plan, a seven-fold difference in how often the code survived it.
That suggests models differ in how much they treat a supplied plan as an order versus a suggestion. A model that quietly deviates from a bad plan looks worse at following instructions and better at repair. If you grade planners and executors on instruction-following alone, you will reward exactly the behavior that makes this failure mode worse. I would want an eval that hands the executor a deliberately flawed plan and measures whether it notices.
The diagnostic checks add one more useful detail. Each misconception family produced its own distribution of violated obligations (Cramér's V of 0.475 on PersistQueue and 0.314 on Dijkstra for plan-only poison), though the mapping was not one-to-one. So a failed run is informative about which wrong idea caused it, which is a cheap way to triage agent failures: look at which contract broke, then ask what plan step would have produced that.
How far I would trust two toy tasks#
Not far on the numbers. This is two synthetic tasks with deterministic execution, and the author says it does not establish a universal effect size. Model variation is huge: the plan-minus-hypothesis gap ranged from −3.8 to +45.1 points on PersistQueue. Claude Haiku 4.5 passed 1.1% of PersistQueue candidates overall while GPT-5.4 Nano passed 67.1%, so baselines differ wildly. 203 requests returned nothing, coverage of family-model combinations is uneven, and no human checked whether a patch actually adopted the planted mechanism. The experiment records what the model was given, not what it believed.
Still, the direction held everywhere it was tested, and it matches how plan-first workflows fail in practice: the agent executes a bad plan competently.
My prediction: the next useful piece of agent tooling is not a better planner but a plan critic that checks the proposed state ownership and update order against executable evidence before any edit lands. Until then I will read plans like code, not like intentions.