Most mornings the agentic-coding arxiv list is wall-to-wall benchmarks, so a single-author observational study nearly slid past me. But its title, reasoning effort not tool access buys first-try reliability, is exactly the kind of claim that cuts against how most of us provision agents right now. We reflexively hand our coding agents more: a browser-testing tool here, a design-oriented system prompt there, on the theory that more capability means better software. This paper ran that assumption into the ground across 90 controlled builds of the same app. My thesis after reading it: the highest-leverage knob on your agent isn't the tool belt, it's the reasoning dial.
What it does#
Reasoning effort, not tool access, buys first-try reliability in agentic code generation (a single-author study by Achint Mehta, posted July 2) does something refreshingly un-benchmark-like. Instead of throwing a task suite at a fleet of models and reporting an aggregate resolve rate, it fixes everything except the variables worth studying. One detailed specification, a real-time retrospective board, the kind of small-but-real collaborative web app with shared state, live updates, and a deploy step, gets built 90 independent times. Every run is scored on the same 14-criterion functional rubric (42 points max) plus a visual-quality review. What varies is the stuff builders actually fiddle with: model generation, which of two agent harnesses, reasoning effort (High versus xHigh), whether a browser-based testing tool is available, and whether a design-oriented system prompt is in play.
The design matters more than it sounds. Because the task is held constant, the study can do criterion-level analysis, not just "run A scored 38, run B scored 31," but which of the 14 things broke, and how often. That turns out to be where the signal lives, because run totals hide it.
The key result#
The headline is a genuinely striking number: raising reasoning effort from High to xHigh lifted first-try perfect runs from 28 percent to 89 percent, more than tripling the odds of a fully correct build on the first shot, while cutting corrective follow-up prompts roughly five-fold. The price of that was 9 to 29 percent more cost. That is an extraordinary exchange rate: a sub-30-percent spend increase buys a roughly 3x jump in first-try success and five times fewer round-trips. Now set it against the testing tool, the thing many of us would reach for first, which raised cost by 42 to 68 percent and improved the functional score by essentially nothing, even on the interface-visible criteria where a checking tool should earn its keep. The design-oriented prompt lifted visual quality (4.5 versus 3.0 on a 5-point scale) but did nothing for function, and a one-paragraph paraphrase of its directive reproduced the entire gain, so the elaborate prompt was mostly ceremony. Read together, the three levers tell one clean story: the lever that changed how the agent reasons moved reliability a lot; the two that changed what it could see or do around the edges moved cost but not correctness.
Why it matters#
If you build or operate coding agents, this reorders your defaults. The instinct when an agent ships broken code is to give it more scaffolding: a test runner, a linter in the loop, a browser it can click around in. Mehta's runs suggest most first-run failures aren't the kind a checking tool catches. They come from weak reasoning about the problem in the first place, and the fix for weak reasoning is a stronger model or more inference effort, not another tool. In Claude Code terms, that's an argument for spending on the reasoning-effort setting (or a more capable model tier) before you bolt a browser-testing MCP onto every sub-agent. Tools aren't free; they cost tokens, latency, and failure surface, and this data says a lot of them buy you nothing on the dimension you actually care about. It also reframes cost: the cheaper-looking config, lower effort with no extra tools, is the one that fails more and spawns more corrective round-trips, so its true cost, your time included, is higher.
The second thing I'm taking is methodological, and it generalizes past this one paper. The container-deployment criterion failed on the first try in 44 percent of runs, and its failure rate swung sharply across model generations, while the mean total score moved less than a point. That's an aggregate score lying to your face: judge these agents by one number and you'd call two model generations equivalent while one of them is silently failing your deploy step half the time. The lesson for anyone designing an agent eval is to score per-criterion and watch the load-bearing ones, does it build, does it deploy, does it start, separately from the cosmetic ones. "Match the fix to the failure" only works if your eval can actually see the failure.
The caveats#
It's a single application, one retrospective-board spec built 90 times, not a diverse task suite, so how far "reasoning beats tools" carries to a 30-file refactor or a long-horizon migration is genuinely unknown.
It's observational and single-author, not a preregistered controlled trial, and the abstract doesn't name the specific models behind "frontier" and "low-cost local," which makes exact replication harder.
"Reasoning effort High versus xHigh" is harness-specific terminology; your mileage depends on what those labels actually do in your stack.
The visual-quality result rests on a subjective 5-point review, and the whole study is one well-instrumented case, not a law.
The takeaway#
What I'm filing away: the reasoning dial is probably the most underused high-leverage knob in the agentic-coding stack, and the reflex to pile on tools is often expensive theater. What I'm doing differently after reading this, before I add another tool to an agent, I'll turn up reasoning effort first and measure whether the tool still earns its cost. And I'm done trusting a single aggregate score to tell me two agent configs are equivalent; the per-criterion breakdown is where the truth was hiding.