← all writing
02 · 17 Nov 2026 · 6 MIN READ

Most multi-agent gains vanish once the single agent gets the same compute

Most of the multi-agent results I've read over the last two years have the same quiet flaw: the team of agents gets five, ten, sixteen model calls, and the single agent it beats gets one.

That's not a comparison of architectures. It's a comparison of budgets. And a new meta-analysis by Seyed Mahmoud Sajjadi Mohammadabadi does the obvious-in-hindsight thing: it pulls 516 multi-agent configurations out of nine published evaluations and re-sorts every result by how strong the single-agent baseline actually was. Once the solo agent gets the same number of samples or the same compute, the multi-agent advantage doesn't just shrink. On average it goes negative.

The baseline everyone beats is the wrong baseline

The paper's core move is a four-rung ladder of baselines. Category A is a single greedy pass. Category B matches the number of samples (self-consistency, majority voting). Category C matches the budget, whether that's thinking tokens or iterations. Category D sweeps the whole accuracy-versus-cost frontier.

Here's how the multi-agent systems fare as the baseline gets more honest:

The first bullet surprised me more than the second. Even against the weakest possible baseline the multi-agent average is slightly negative. Drop the single study that contributes 180 of the 516 configurations and it lands near zero (−0.005). Either way, the "agents beat a single model" story mostly lives in individual papers, not in the pooled evidence.

Five samples and a vote, no conversation

The cleanest illustration comes from one of the included studies (Choi et al.), across 14 model-and-benchmark cells:

Debate does beat the greedy pass by 4.5 points. But five independent samples with a vote beat it by 7.6, and with no talking at all. The communication didn't just fail to add value. It cost 3.1 points relative to doing nothing clever with the same five calls.

The author calls the difference between "versus greedy" and "versus sampled" the sampling gap, and it is negative in all five studies that report both baselines, averaging −0.038. Put plainly: a good chunk of what gets credited to coordination is just the variance reduction you get from drawing more than one sample. The token-matched results point the same way. In one study, once both sides had 500 or more thinking tokens, the single agent matched or beat debate, with the difference running −0.023 to −0.035 from 1,000 tokens upward.

The paper's explanation is information-theoretic and, I think, correct in spirit:

Because communication adds no information when agents share identical evidence, positive returns require partitioned evidence, external verification, or error diversity across models.

If every agent reads the same prompt, a message from agent B is just a lossy re-encoding of what agent A could have computed itself. You're paying for the round-trip.

Software is where the exception clause lives

This is where it gets relevant for anyone building coding agents, because two of those three escape hatches are things our domain has for free.

External verification. The paper's third decision rule is literally "enforce external verification against ground truth," and the examples it names are compiler checks, unit tests and database queries. A coding sub-agent that runs the test suite is not trading opinions with its peers; it's pulling new information into the system. That's a categorically different thing from a reviewer agent that reads the diff and says "looks good."

Partitioned evidence. The paper's second rule says teams are justified when inputs exceed one model's context, when tools are independent and parallel, or when execution states are sandboxed. That describes a large monorepo search fanned out across sub-agents pretty well. It does not describe a planner, a coder and a critic all reading the same issue text.

The software-specific numbers in the corpus are thin but worth noting. In the iteration-matched study by Kim et al., the mean difference on SWE-bench Verified was −0.017 and on Terminal-Bench −0.067. That study also reports gains on decomposable financial tasks but penalties on sequential software planning, with multi-agent gains turning negative once the base model clears roughly 45% accuracy. The one big positive outlier is telling: on gpt-5-nano, the multi-agent setup went from 5% to 35%. Weak model, low floor, and the scaffolding carries it. Strong model, and the scaffolding becomes overhead.

And there's a striking non-coding data point on what verification-grade structure buys you. On one benchmark (SMFR, with GPT-5), automated multi-agent frameworks scored between 20.2% and 63.0% at costs from $427 to $9,239. Plain chain-of-thought with self-consistency hit 57.0%. A human-built expert tooling system reached 96.51% for $555. One benchmark, one specialized system, as the paper says itself. Still, the pattern is hard to miss: structure grounded in the task beat generic agent chatter by a mile.

Nine studies, one of them doing most of the talking

I'd hold the exact numbers loosely. The author is upfront about the weaknesses: the first five studies were picked without a search protocol, one paper supplies 180 of the 516 configurations, means are unweighted, and some contrasts rest on small test sets (Kim et al.'s software contrasts are 20 instances each). Most of the corpus is reasoning and QA benchmarks like MMLU-Pro, MuSiQue and FRAMES, not repository-level coding. And none of the nine studies controls latency, which is the cost I feel most when a pipeline runs three sequential agent rounds.

There's also a counterexample in the corpus: on MMLU-Pro at matched compute, debate and mixture-of-agents beat self-consistency by 1.3 and 2.7 points. So "multi-agent never helps" isn't the finding. The finding is that the burden of proof was placed on the wrong side.

My rule for sub-agents from here on

When I spin up sub-agents in Claude Code or in my own orchestration, I'm now asking one question first: what does this agent see or check that the parent can't? If the answer is "a different file range", "a sandboxed test run" or "the real API response", it earns its slot. If the answer is "the same context, with a different system prompt telling it to be critical", I'll try five samples and a test-based pick from the parent instead and compare at the same token spend.

The paper's PACE checklist (report thinking tokens, total tokens, dollars, latency, sample count and tool scaffolding for every comparison) is something I'd adopt for internal evals verbatim. Most agent dashboards I've seen log maybe two of those six.

My prediction: within a year the multi-agent coding systems that hold up will be the ones whose agents are defined by what they can execute, not by what persona they're told to play.


Working on something similar?

Say hello — I read every email.