← all writing
02 · 21 Nov 2026 · 6 MIN READ

Losing the top coding model for 19 days didn't dent anyone's merge rate

My first reaction to this paper was that it had to be wrong. Every model release I've lived through felt like a step change in what I could hand to an agent. So a study claiming the newer, higher-scoring model made no measurable difference to finished work set off every skeptic alarm I have.

Then I read how they got there, and most of my objections were already answered in the appendix.

The paper is Higher-Scoring Coding Models Did Not Get More Work Done by Yining Hua (Harvard). The setup is a genuine natural experiment: on June 12, 2026, a U.S. export control directive cut off access to Claude Fable 5 for 19 days. Fable 5 led Opus 4.8 by 10.8 points on SWE-Bench Pro. Same developers, same projects, and suddenly a different model, not by choice. Hua tracked 297 Fable 5 users through the cutoff and back, against 8,908 developers who stayed on Opus 4.8 the whole time.

A forced downgrade is about as clean as field data gets

My usual complaint about productivity studies is confounding. Early adopters are better engineers. People route hard tasks to the strong model and easy ones to the cheap one. A handful of repos with lax merge habits dominate the sample. This design sidesteps most of that, because the intervention came from outside and hit everyone on the same day.

Model identity comes from the Co-Authored-By line Claude Code adds to commits, when it also ends up in the PR description. A check of 200 PRs found the description matched the commits 96 to 99% of the time. The analysis is difference-in-differences with developer and calendar-day fixed effects, standard errors clustered by developer. The behavioural switch was sharp: 82.3% of the treated group's PRs named Fable 5 while it was available, 0.9% during the suspension, with 74.6% naming Opus 4.8 instead.

Here is what moved:

The upgrades are the part that convinced me more than the suspension. Six of them came with SWE-Bench Pro gains of 5 to 18 points. Five of the six merge-rate point estimates were negative. None were significant. When the same developers switched within the same repository, the merge rate actually fell by 1.2 points, and that one was significant.

My objection was the ceiling, and it's a real one

The obvious attack: if almost every PR already merges, the merge rate has nowhere to go. And that's true. In the week before each upgrade, developers who switched had 94.5% of PRs merged within seven days, and 83.8% of them had every single PR merged.

Hua anticipated this. Merging within one hour started around 75%, plenty of headroom, and didn't move either (0.0 points for the suspension, +0.8 for upgrades). Among developers who did have unmerged PRs before an upgrade, the switchers' rate changed by 1.3 points less than non-switchers. Throughput per week didn't rise. Neither did check failures, which is the outcome closest to code quality the data allows.

So the ceiling explains part of the flat line, but not the whole story. What bothers me more is what a merge even means here:

95% were merged by the author, and 90% had no approval from any other account

Median time from open to merge was under half an hour in every phase, for every model. This is not code review. It's a developer deciding the agent's work is done and clicking the button. The paper is upfront about it: if model differences mainly show up when someone else reviews the code, the estimates are too small. I'd go further. This metric measures "the human accepted it," and humans accept work they've already scoped to be acceptable.

A few other honest limits worth naming. Only about 1% of merged Claude Code PRs name the model in the description, so the sample is whoever leaves that line in. The suspension estimate rests on roughly 7% of the treated developers' PRs, though a refit on all their PRs points the same way (+0.5 points). The restoration phase collides with the Sonnet 5 release, the July 4 holiday, and a usage cap on Fable 5. And the study can't see time saved per PR, only outcomes.

Developers were handing off work the old model could already do

Here's where I landed: partly convinced, and the part I'm convinced of is more useful than the headline.

Hua's own hypothesis is that the tasks people delegate already sit inside what the older model can handle. The paper cites a survey of 132 engineers and researchers at Anthropic: they mostly gave Claude tasks that are easy to check and cheap to get wrong, and more than half could fully hand off only 0 to 20% of their work. Crucially, developers did not start giving the new model harder work: PR description length, files mentioned, and the share of feature PRs didn't change after upgrades.

That's the finding I'd underline. The bottleneck isn't the model's ceiling. It's the delegation boundary we draw, and we don't redraw it when the model changes.

It also matches what I see in my own Claude Code sessions. I scope tasks to what I trust from the last model, write the spec at the granularity that worked last month, and review at the depth that habit dictates. A new model lands, and I keep feeding it the same-shaped tickets. Of course the merge rate doesn't move. I never gave it a chance to.

What I'm changing

Three things, mostly about my own evals rather than vendor claims.

First, when a new model ships, I'll pull a handful of tasks I previously refused to delegate, the vague, cross-cutting ones, and run those instead of my usual queue. Benchmarks measure capability; my backlog measures my habits. If the frontier moved, it moved on the work I'm not handing off.

Second, I'm dropping "PRs merged" as a success signal for agent tooling decisions entirely. With 90% of merges unapproved by anyone else, it's a measure of my own confidence, not of the code. Revert rate, follow-up fix PRs, and review comments from someone who didn't prompt the agent are slower to collect but actually informative.

Third, harness and spec work deserves more of the budget I'd otherwise spend chasing the newest model. If the delegated work already fits inside the older model, the leverage is in widening what I delegate safely: better specs, tighter verification, clearer escalation paths.

The open question I'd love someone to answer with this same natural-experiment trick: does the gap appear once you restrict to PRs reviewed by a second human? My bet is yes, modestly. But until someone measures it, an 11-point benchmark lead is a claim about tasks most of us aren't giving our agents.


Working on something similar?

Say hello — I read every email.