Skimming Friday's cs.SE list I almost scrolled past this one: single author, no lab affiliation, a title that reads like a methods footnote. Then I hit the number. 48.8% — fewer than half the MCP servers in a random draw from the official registry will even complete an initialize handshake. Every ecosystem-health study I had read before this one sampled servers that already worked. The thesis of this post: the MCP ecosystem you have been reading about is a curated highlight reel, and the tool-use benchmarks you trust are mostly duplicates.
What it does#
What a Random Draw from the MCP Registry Contains, and What Tool-Use Benchmarks Contain Instead by Haseeb Mohammed Afsar does something almost nobody does in ecosystem papers: it draws a probability sample and then refuses to fix anything. Full census of the official registry on two dates, July 14 and August 22, 2026 — 24,135 unique servers on the second sweep. Filter to npm-published, stdio-declared, active servers: 7,258 candidates. Draw 400 without replacement via Fisher–Yates off a seeded mulberry32 generator (seed 20260819), and record the SHA-256 of the frame bytes so frame drift is detectable after the fact.
Then probe each one over the wire. An npx launch, one attempt, a purpose-built prober speaking the raw MCP protocol: initialize handshake, tools/list, JSON Schema validation on every advertised tool. No credentials. No side-effecting tool invocation. No server repair. No retries. That last part is the whole contribution. Prior work samples from reference sets, popularity lists, hand-curated frames, or pipelines that patch a server until it boots — every one of those quietly conditions on "works." This paper records the failure instead of engineering around it.
The key result#
195 of 400 servers — 48.8% — completed the handshake. The same instrument pointed at a hand-curated frame of reference and popular servers returned 66.7%, a 17.9-point gap that is pure curation effect. And the dominant failure is not the one I would have guessed: missing credentials accounted for only 13.3%, while 37.5% — 150 servers — simply never started at all. Two more were unavailable on npm. The second half of the paper is sharper. Holding one method constant across corpora (TF-IDF, cosine, global exact dedup enforced first), real MCP tools show 2.8% near-duplication at cosine 0.70, and all of it sits inside single servers — cross-author near-duplication is 0.0% at every threshold tested. BFCL v4 shows 16.7%, of which 16.4 points lie between independently presented tasks. And before deduplication: 68.8% of raw BFCL rows and 85.6% of raw UltraTool rows are exact name-plus-description repeats, against 0.4% for real MCP.
Why it matters#
If you are building an agent that pulls MCP servers dynamically, your failure budget is wrong. You have probably been designing for auth failures — token missing, scope wrong, expired — because that is the failure that shows up in docs and in your own testing against servers you hand-picked. The real distribution is close to the inverse: roughly three times as many servers die at launch as die at auth. That changes the shape of the code. A retry-with-credentials path is worth less than a hard process timeout, a clean "this server is dead" terminal state, and a health signal you maintain yourself rather than trusting the registry. Reinforcing that: zero of 2,766 tools carried a fatal JSON Schema violation, and zero of 195 servers had any. Hard conformance among servers that run is total. So the schema-validation layer I would have written first is mostly wasted effort, and the process-supervision layer I would have skipped is exactly where the failures live.
The annotation finding is the one that actually changed a design decision for me. 58.8% of tools carry no safety annotations at all — no readOnlyHint, destructiveHint, idempotentHint, or openWorldHint — and the distribution is bimodal, not gradual: 72 of 194 servers annotate every tool, 122 annotate none, and zero servers sit in between. You cannot build a "block destructive tools" gate on top of annotations, because for more than half the ecosystem there is nothing to read. You need your own classifier or an explicit allowlist. On the eval side: if you have been quoting BFCL v4 as evidence that a model handles tool use, 16.4 points of near-duplication between independently presented tasks means part of that score rewards memorizing a repeated distribution. The paper is careful here — UltraTool shows only 0.3% cross-task redundancy, cleaner than real tools, so this is a property of BFCL specifically and not of synthetic corpora as a class. The structural point survives either way: real tool ecosystems are lexically diverse across authors because independent vendors never converge on naming, and an agent tuned only on benchmark tools has seen a far narrower description space than production will hand it.
The caveats#
Scope is narrow and shrinking. npm/stdio only, which was 30.7% of the population at the second sweep, down from 35.1% just 39 days earlier while remote-only servers grew 70.1%. The paper measures the slice of the ecosystem that is being abandoned.
One probe, no retries is methodologically clean but conflates permanent breakage with transient flakiness — these are cold npx installs run under concurrency with a kill timeout. The authors are explicit that their wall-clock numbers are not a latency measurement, but I would want a two-attempt variant before treating 37.5% as a permanent-death rate.
TF-IDF cosine is lexical, not semantic, which biases every duplication rate downward — BFCL's included. The direction is honest, but every magnitude here is a floor.
Unit mismatch. BFCL averages 1.8 tools per task against 14.2 per server for real MCP. Capping real MCP at 10/25/50 tools per server yields 0.9%/2.2%/3.0%, which brackets the uncapped 2.8% — reassuring, but a construct difference you cannot fully wash out. And two snapshots support a difference, not a trend.
The takeaway#
What I am filing away is the method more than the numbers. "Sample without repairing" is a discipline I do not apply to my own agent evals nearly enough — every time I debug a flaky tool until it works and then count it as working, I am quietly building a curated frame and then believing it. Concretely, two changes: I am putting a process-liveness gate ahead of the schema-validation gate in anything that loads MCP servers dynamically, and I have stopped treating tool annotations as a safety surface I can rely on. And the next time I am about to cite a tool-use leaderboard, I am checking its deduplication before I quote the number.