Skimming this morning's arxiv list I hit the experiment I've been waiting a year for someone to actually run: a controlled MCP-versus-CLI cost comparison, with the task held fixed and completion verified by inspecting the repository rather than asking the agent how it went. I expected a number that would finally settle the argument. What I got was more useful — the argument is about the wrong variable. The thesis of this post: your tool interface is a rounding error next to the harness driving it.
What it does#
The Scaffolding Matters More Than the Interface (Alier Forment, Casañ Guerrero, García-Peñalvo and Pereira — UPC Barcelona, Salamanca, UPV/EHU) fixes one task and varies three things independently: the agent scaffolding, the language model, and the tool interface. The task is six operations against a private GitHub repository — locate an open issue, create a branch, apply a supplied patch, commit it, open a pull request, and report how many files a named directory contains. Seven scaffoldings: Claude Code, OpenAI Codex, qwen-code, Hermes and opencode, plus pi and Tau — two minimal agents that ship no MCP client at all and serve as controls. Five models: two hosted by OpenAI, one by Anthropic, and two open-weight models served on local hardware. Fifty-four cells in the main matrix.
Two design choices carry the paper. First, completion is verified by querying the GitHub API for four independent conditions — branch exists, file contains the patched content, PR opened, file count correct — and never by the agent's own report. A run that declares success and leaves no branch behind scores nothing. Second, the arms are isolated by credential, not instruction: on the command-line arm no MCP server is configured anywhere, and on the MCP arm the shell is pointed at an empty credential directory, so any quiet fallback to gh fails visibly instead of succeeding unnoticed. Every request routes through a local proxy, so scaffoldings that delegate to sub-agents can't under-report by hiding consumption. The harness, task definition, verifier and run-by-run dataset are all published.
The key result#
Same task, same models, same verification — only the harness driving the model changes — and the median input-token cost of a completed run spans a factor of twenty: 14,660 tokens for pi and 16,459 for Tau, against 260,170 for Claude Code and 288,808 for qwen-code. Restrict to the cleaner slice where no MCP catalogue is attached to anything and the two scaffoldings without an MCP client come in 5.0x to 28x cheaper than the five widely used ones doing identical work. Meanwhile the comparison the authors set out to make came out inconclusive: thirteen strictly paired MCP-to-CLI ratios span 0.43x to 29.06x with a median of 0.93, against a measured run-to-run variation of 1.51x — anything under roughly twofold is indistinguishable from noise. Output tokens barely moved at all: 824 to 1,367 across all seven scaffoldings, a factor of 1.7 while input varied by 20.
Why it matters#
The paper measured something I've never seen reported and now want on every agent dashboard: how many tool schemas each scaffolding transmits per request. Four of the five MCP-capable tools send the GitHub server's complete catalogue — all 44 tool descriptions — with every single request, so a task taking twenty exchanges pays for that catalogue twenty times. Hermes sends seven, of which two are a gateway pair (one tool that lists what's available, one that invokes a tool by name); the other 42 descriptions are fetched only when the model asks. Median MCP-arm cost: 70,836 tokens versus 216,986 — a factor of 3.1, same protocol, same server, same operations exposed. And that's before your own tools: Claude Code sends 74 schemas on its MCP arm (27 built-in, 3 for reading MCP resources, 44 from the server), and its command-line arm still carries the 27, roughly 22,000 tokens spent before the task has said anything. Lazy delivery is a design choice available to you, not a protocol constraint — Anthropic's own code-execution-with-MCP guidance points the same direction.
The split the evidence actually lands on is general-purpose versus purpose-built. The five popular scaffoldings are built for open-ended work and carry the machinery breadth requires on every task — including a task that needs a shell, three file operations and one credential. For exploratory work that breadth is what you're paying for. For a well-specified job that runs on a schedule, or in a pipeline, or forty times a day, it's pure overhead, and the effect compounds downward: a 27-billion-parameter local model completed this task under every scaffolding it was given, at costs ranging from 17,416 tokens to 2,418,828 — a 139x spread on the same model doing the same work. The model was never the constraint. Small models have the least context to spare, so they pay the generality tax at the highest rate. If you've been waiting on better small models to make self-hosting viable, the more actionable read is that the factor of 139 lives in the software around them.
One more finding with operational teeth. In a companion experiment on a public repo, with the MCP server attached and confirmed connected in 21 runs: six agents used MCP tools exclusively, six completed the task entirely through the shell without calling an MCP tool once, six used both, three called no tools at all, and four bypassed both interfaces and hit the GitHub web API directly over HTTP. Adding a prompt instruction naming the assigned interface didn't change this materially; removing the credentials for the alternative did. As the authors put it, a measurement that assigns an interface without verifying which one was used reports the cost of an unknown mixture. The same applies to your production traces: if you assume your agent uses the tools you handed it, go check.
The caveats#
One task, one domain. GitHub has both a mature CLI and an official MCP server. A service publishing MCP operations with no command-line equivalent would favour MCP far more than this task can show.
The controls differ in more than MCP support.
piandTauare compared as a population against the other five, not as an isolation of one feature — the authors say so explicitly rather than overclaiming.The delivery-method result rests on one scaffolding. Hermes is the only tool in the sample that fetches descriptions on demand, so "lazy delivery" is confounded with everything else that makes Hermes different. They flag it as a mechanism worth testing, not one established.
Thin sampling on the hosted rows. Most hosted configurations ran once, for budget reasons, against a resolution limit of about 2x. Claude Code's row rests on six live runs and includes the priciest model in the study — read that 17.7x loosely. Its MCP arm was actually cheaper than its CLI arm, at 0.57x.
The authors have a prior thesis this supports. Their earlier methodology paper argues for small purpose-built agents. To their credit they disclose it, fixed the task and verification before running the matrix, and published the inconclusive interface comparison they had hoped would be decisive.
The takeaway#
Four numbers I'm filing: 20x between scaffoldings on one identical task, 139x on a small local model, 6x in schemas transmitted per request for the same protocol against the same server, and an interface effect that never clears the noise floor. What I'm doing differently: before I touch an MCP config again, I'm measuring what my harness costs at rest — schemas per request, tokens burned in the first turn before the task begins — and I'm checking which tools my agents actually called rather than which ones I gave them. For the scheduled, well-specified jobs I run over and over, the case for a small scaffolding I can read in an afternoon just got a price tag attached to it.