---
title: "Does Multi-Model AI Coordination Beat a Single Model? A Coding Benchmark"
canonical: "https://agentictool.ai/posts/does-multi-model-coordination-beat-a-single-model"
source_html: "https://agentictool.ai/posts/does-multi-model-coordination-beat-a-single-model.html"
description: "The finding: higher floor, lower ceiling. We ran nine ways to combine AI models on 49 real coding problems. None beat the single best model, and the missing frontier baseline we added after review widened the gap. Coordination lifted a cheap model by 9 and reached about 86% of the frontier with no model-picking, while capping the top end. Here is exactly when teaming up wins, and when to just use one model. Every number is a receipt."
date: 2026-07-08
---

ATO · multi-model coordination benchmark · receipts inside

# Does multi-model AI coordination beat a single model?

**The finding: higher floor, lower ceiling.** We ran **nine ways to combine AI
 models** on 49 real coding problems. None beat the
 single best model, and when we added the frontier baseline our reviewers asked for, the gap got
 *wider*. Coordination **lifted a cheap model by 9** and reached
 **about 86% of the frontier with no model-picking**, while capping the top end. Here is exactly when
 teaming up wins, and when to just use one model. Every number is a receipt.

+9problems coordination adds to a cheap model (28 to 37)
 86%of the top model's score, with no model-selection
 43/49the best single model. No team beat it
 4,297metered API calls. Every cost is a receipt

Everyone sells multi-model "orchestras" that supposedly beat any single model.
 We tested the claim head-on. Nine models solo, nine ways of combining them, contamination-audited,
 graded by running the code, with every result on the table. The truth is more useful than the hype.

Prefer the rigorous version?
 The full scientific paper is a **draft**.
 [**Preview it**](https://agentictool.ai/posts/coordination-benchmark-paper)
 or [**download the PDF**](https://agentictool.ai/posts/coordination-benchmark-paper.pdf)
 (watermarked draft): methodology, statistics, all three figures, and references.

## The hypothesis, and what we actually found

We expected a group of models to answer *more* questions than the best single model. It did not.
 The best single model solved **43/49**; the best team solved **38**. (Nine recipes total: five ran
 the full 49 tasks, four ran a 12-task pilot.) But coordination was not
 useless. It did something different and genuinely useful. Picture a **study group** taking the test.
 It is far better than the weakest student, because it catches their mistakes, but a touch worse than the
 smartest one, because it sometimes argues them out of a right answer. Coordination **raises the floor
 and lowers the ceiling.**

## Update, 2026-07-10: we ran the missing baseline. The headline got stronger.

Reviewers of the draft paper flagged a real gap: our two
 escalation recipes call **claude-fable-5** as the strong closer, but we never benchmarked fable solo.
 Fair hit. We ran it. **Fable solo scored 43/49**, the new best single model, and its six misses are
 almost exactly the tasks *no model or team has ever solved* (five of six). Two consequences, both
 receipts-on-the-table honest. First, the headline claim survives and widens: the best team still solved
 38, so the orchestra now trails the best soloist by five, not three. Second, and more interesting: the
 flagship cascade (cheap model, escalate to fable on failure) is **strictly dominated** by just calling
 fable every time. Across all 49 tasks the cascade never solved one that fable solo missed, and it lost
 five that fable would have gotten (discordant 5:0, exact McNemar p=0.0625; suggestive, not significant,
 and we say so). Why? On four of the five, the cascade only escalates when the cheap model *fails the
 public tests*, and the cheap model passed the public tests with code that failed the hidden ones, so
 the strong closer was never called. On the fifth the gate fired and the closer's answer still died in the
 sandbox (an `import numpy` the isolated grader does not provide). The bottleneck is rarely the
 strong model. It is the **verifier that decides when to call it**. That is the single most useful engineering lesson in this study, and it cost us one review to
 find. Two disclosures: fable's training cutoff (2026-01) postdates every task, so like sonnet it is
 contamination-exposed; among contamination-clean models the best single remains **gemini-3-flash at
 41/49**, still ahead of every team. And 35 of fable's 49 tasks ran through the Claude subscription CLI
 rather than the metered API, so its cost figure is anchored on the 14 metered tasks (receipts flag the
 channel per task). The [draft paper](coordination-benchmark-paper.html) is revised accordingly
 (2026-07-10): a fable row in Table 1, a per-model contamination audit, and a new Section 4.4 on the
 verifier bottleneck.

**Read one receipt yourself.** Task 3696:
 the cheap model's code passed the public tests (primary_public_passed: true), so the gate
 never escalated (escalated: false), and the hidden suite failed it 4/43. The strong closer
 was paid for and never called; fable solo solves it. Task 3701 is the one that breaks the
 pattern: the gate fired (escalated: true), and the closer's in-cascade answer died on an
 import numpy the isolated sandbox does not provide, while its solo answer used the standard
 library. Both receipts, and every other number on this page, are in the
 [published replays](https://app.agentictool.ai/?replays): the routing decision, the gate flags, the grader
 verdict, per task. That is what open-box means.

**single model
 **coordination recipe
 **coverage ceiling (perfect pick)
 
 Cheaper is left, better is up. The mint line at 44 is the **coverage ceiling**, what
 you would hit if you could always pick the right model per task. No recipe reaches it, because picking
 is hard. Hover any point for its exact cost and score.

## Raising the floor, the number that matters

A cheap model alone solves **28/49**. Wrapped in a coordination recipe it solves **37 or 38**,
 a lift of 9 to 10 problems, about 32% on its own score. Framed as a bet: pick a cheap model at random
 and you would average 59%; coordinate a handful and you reliably get **76%**, which is **86% of the
 top model's score, with zero model-picking skill required.** Coordination is **insurance against
 picking the wrong model.** (Fair print: that five-model group contains deepseek-reasoner, which alone
 scores 39/49 at a ninth of the group's cost. The insurance framing holds only when you cannot tell which
 of your cheap models is the good one.)

The same cheap model, alone versus inside a team: **28 to 38**. The team catches the
 weak model's mistakes.

## Lowering the ceiling, the honest half

The catch shows up only in **symmetric "war rooms"** (models debating as peers). There, the group
 can score below its own best member. Our five-model war room scored **37**. One of its own members,
 **deepseek-reasoner, scores 39 alone**. That is a cheap model, so this is a fair, same-tier
 comparison. On three specific tasks the group got a problem *wrong* that that same model got
 *right* by itself. The group talked it out of the correct answer. Asymmetric recipes, where a cheap
 model escalates to a strong one, do not pay *this* penalty; nobody argues the strong model out of
 its answer. But the fable baseline revealed they pay a different one: the cascade escalates only when the
 cheap model fails the *public* tests, and code that passes public tests while failing hidden ones
 never gets escalated. That gate cost the cascade five problems its own closer would have solved. The
 war room's tax is social; the cascade's tax is a weak verifier.

Same model, alone versus inside the war room: **39 to 37**. Consistency up, peak down,
 the classic ensemble trade. This is one n=49 run, so read the drop as directional, not proven.

Coordination gains about 9 problems at the floor and gives back 2 to 5 at the ceiling.
 Versus your best pick you lose. Versus a blind
 pick you gain a lot.

## So when do you team up, and when do you just use one model?

**Team up** when you cannot run or afford the top model, when you do not know which model is best
 for a task, or when a rare catastrophic answer is unacceptable. The group is the safe bet.
 **Use one model** when you know and can afford the best one, because it scores higher, costs less, and
 is simpler. **Coordination is insurance for when you cannot pick. Routing is knowing which single model
 to pick.** ATO's job is to tell you which situation you are in, with receipts, per task.

## The asymmetric case works, and Anthropic just showed it

One shape wins on cost: a cheap model that consults a strong one only when needed. Anthropic's own
 advisor tool shows it. On long coding tasks, a cheap executor calling a flagship advisor about once per
 task hits **about 92% of the flagship's score at about 63% of the cost.** Real, and it matches our
 finding. The catch is that it depends on how often you call the expensive model. Rare, on long tasks, is
 cheap. Constant, on short tasks, is not. That is a per-task routing call. And Anthropic's version routes
 only among Anthropic models, while the open version routes across *all* your providers.

## This is one experiment. There are many more coming.

The **pick** is where the real gains hide, and we tested
 it two ways. First a strong coordinator that **grills the candidates, runs their code, and decides**
 picked the correct answer 12 of 12 on the hardest contested tasks, but it was strong enough to solve them
 itself, so that was only suggestive. Then the clean test: the **weakest model in the study, which solves
 just 2 of those 12 alone, still picked the correct answer 9 times, and 7 of 10 on tasks it could not solve
 at all.** Recognizing the right answer looks easier than producing it, even for a weak model. Honest
 caveat: this is a **12-task probe, single sample**, so we treat it as a strong lead to confirm, not a
 settled result.

**What we are testing next, two hypotheses.** First, that
 recognition result **at full scale**, many judges, many tasks, repeated samples, to see whether a weak
 model can reliably pick answers it cannot produce. Second, and the bigger one: this study is single-shot
 competitive programming, the adversarial case for coordination, so surviving it at all is respectable. The
 real test is **long-horizon agentic work with complementary roles**, one model reasons, another
 generates, a third verifies, over real multi-step loops. If coordination beats the best single model
 anywhere, it is there. Both ship with the same receipts, and the methods that win become **ATO's Pro
 coordination recipes.**

Here is the deal. When you support ATO, you are funding
 **real, open coordination research**, run on real keys, with real receipts, and
 **shared with the community.** Not a vendor's marketing numbers you cannot reproduce, but
 experiments you can re-run and build on. **Back the tool, fund the research, get the receipts.**

## Don't trust our number. Get the receipts.

Three ways in. All open, all reproducible. 49 tasks is not
 gospel, and that is the point: run it on your own models.

Free

### Run it yourself, on any models you have keys for.

Download ATO, add your keys, run one line. Any model set you want, no charge. You do the steps.

`ato bench run --models "gpt-5,gemini-3-flash,your-model" --suite coordination`
 [Download ATO, free](https://agentictool.ai)
 

 
 Free account

### See our receipts. Replay the war rooms.

Log in and watch the actual transcript that produced these numbers, with every receipt. This is
 the open box Fugu will not show you.

[See the receipts](https://app.agentictool.ai/?replays)
 

 
 Pro

### Get the button.

Let ATO run the benchmark on your models on a schedule, and apply the routing pick automatically.
 The effort is free. The button is Pro.

[Get the automation](https://agentictool.ai)

On the honesty ledger: this is 49 tasks, run once at temperature 0 (with a fourfold
 resample on the contested ones). That is enough to be directional, not a universal law, which is exactly
 why the tool ships the suite instead of just the claim. The "44/49 ceiling" is a *coverage* number,
 meaning what two models could solve if you always picked right, not a score any recipe reached. Costs:
 seven of the nine solo rows and every metered recipe ran on per-token API keys, so those costs are
 receipts, not guesses. Two rows went through the Claude subscription CLI (the sonnet row, and 35 of
 fable's 49 tasks): grading there is identical, but the CLI does not bill per call, so those cost cells are
 API-equivalent estimates and are flagged as such in each receipt. Fable's headline cost comes from its 14
 metered tasks. Contamination is per-model, not blanket: gemini, gpt-5, and o3-mini rows are
 contamination-clean (tasks postdate their training cutoffs); the two Claude models have cutoffs after the
 task window, so they are contamination-exposed; DeepSeek publishes no cutoff, so unknown. The clean-model
 headline is gemini's 41/49, and no team beat that either. Your keys, your models, your numbers.

**Why these four labs.** We tested models from OpenAI, Anthropic, Google, and DeepSeek
 because they are among the best labs shipping right now on public coding and reasoning tests, they include
 both the leading closed providers and a leading open-weight one (DeepSeek), and together they cover the
 whole price range a real developer routes across, from a frontier model at about ten cents a task down to
 a cheap one at a fraction of a cent. We also tried adding z.ai (the GLM models), but it kept draining our
 API credit far faster than the usage it reported, cents on paper while dollars left the balance. We reached
 out to z.ai to sort it out and got no response, so we could not get it working for the coordination runs.
 Since this whole project lives or dies on trustworthy receipts, we stopped testing it rather than publish a
 cost we could not verify. That is our own experience with our setup, not a verdict on the provider.