The finding: higher floor, lower ceiling. We ran nine ways to combine AI models on 49 real coding problems. None beat the single best model, and when we added the frontier baseline our reviewers asked for, the gap got wider. Coordination lifted a cheap model by 9 and reached about 86% of the frontier with no model-picking, while capping the top end. Here is exactly when teaming up wins, and when to just use one model. Every number is a receipt.
Everyone sells multi-model "orchestras" that supposedly beat any single model. We tested the claim head-on. Nine models solo, nine ways of combining them, contamination-audited, graded by running the code, with every result on the table. The truth is more useful than the hype.
Prefer the rigorous version? The full scientific paper is a draft. Preview it or download the PDF (watermarked draft): methodology, statistics, all three figures, and references.
We expected a group of models to answer more questions than the best single model. It did not. The best single model solved 43/49; the best team solved 38. (Nine recipes total: five ran the full 49 tasks, four ran a 12-task pilot.) But coordination was not useless. It did something different and genuinely useful. Picture a study group taking the test. It is far better than the weakest student, because it catches their mistakes, but a touch worse than the smartest one, because it sometimes argues them out of a right answer. Coordination raises the floor and lowers the ceiling.
Reviewers of the draft paper flagged a real gap: our two
escalation recipes call claude-fable-5 as the strong closer, but we never benchmarked fable solo.
Fair hit. We ran it. Fable solo scored 43/49, the new best single model, and its six misses are
almost exactly the tasks no model or team has ever solved (five of six). Two consequences, both
receipts-on-the-table honest. First, the headline claim survives and widens: the best team still solved
38, so the orchestra now trails the best soloist by five, not three. Second, and more interesting: the
flagship cascade (cheap model, escalate to fable on failure) is strictly dominated by just calling
fable every time. Across all 49 tasks the cascade never solved one that fable solo missed, and it lost
five that fable would have gotten (discordant 5:0, exact McNemar p=0.0625; suggestive, not significant,
and we say so). Why? On four of the five, the cascade only escalates when the cheap model fails the
public tests, and the cheap model passed the public tests with code that failed the hidden ones, so
the strong closer was never called. On the fifth the gate fired and the closer's answer still died in the
sandbox (an import numpy the isolated grader does not provide). The bottleneck is rarely the
strong model. It is the verifier that decides when to call it. That is the single most useful engineering lesson in this study, and it cost us one review to
find. Two disclosures: fable's training cutoff (2026-01) postdates every task, so like sonnet it is
contamination-exposed; among contamination-clean models the best single remains gemini-3-flash at
41/49, still ahead of every team. And 35 of fable's 49 tasks ran through the Claude subscription CLI
rather than the metered API, so its cost figure is anchored on the 14 metered tasks (receipts flag the
channel per task). The draft paper is revised accordingly
(2026-07-10): a fable row in Table 1, a per-model contamination audit, and a new Section 4.4 on the
verifier bottleneck.
Read one receipt yourself. Task 3696: the cheap model's code passed the public tests (primary_public_passed: true), so the gate never escalated (escalated: false), and the hidden suite failed it 4/43. The strong closer was paid for and never called; fable solo solves it. Task 3701 is the one that breaks the pattern: the gate fired (escalated: true), and the closer's in-cascade answer died on an import numpy the isolated sandbox does not provide, while its solo answer used the standard library. Both receipts, and every other number on this page, are in the published replays: the routing decision, the gate flags, the grader verdict, per task. That is what open-box means.
A cheap model alone solves 28/49. Wrapped in a coordination recipe it solves 37 or 38, a lift of 9 to 10 problems, about 32% on its own score. Framed as a bet: pick a cheap model at random and you would average 59%; coordinate a handful and you reliably get 76%, which is 86% of the top model's score, with zero model-picking skill required. Coordination is insurance against picking the wrong model. (Fair print: that five-model group contains deepseek-reasoner, which alone scores 39/49 at a ninth of the group's cost. The insurance framing holds only when you cannot tell which of your cheap models is the good one.)
The catch shows up only in symmetric "war rooms" (models debating as peers). There, the group can score below its own best member. Our five-model war room scored 37. One of its own members, deepseek-reasoner, scores 39 alone. That is a cheap model, so this is a fair, same-tier comparison. On three specific tasks the group got a problem wrong that that same model got right by itself. The group talked it out of the correct answer. Asymmetric recipes, where a cheap model escalates to a strong one, do not pay this penalty; nobody argues the strong model out of its answer. But the fable baseline revealed they pay a different one: the cascade escalates only when the cheap model fails the public tests, and code that passes public tests while failing hidden ones never gets escalated. That gate cost the cascade five problems its own closer would have solved. The war room's tax is social; the cascade's tax is a weak verifier.
Coordination gains about 9 problems at the floor and gives back 2 to 5 at the ceiling. Versus your best pick you lose. Versus a blind pick you gain a lot.
Team up when you cannot run or afford the top model, when you do not know which model is best for a task, or when a rare catastrophic answer is unacceptable. The group is the safe bet. Use one model when you know and can afford the best one, because it scores higher, costs less, and is simpler. Coordination is insurance for when you cannot pick. Routing is knowing which single model to pick. ATO's job is to tell you which situation you are in, with receipts, per task.
One shape wins on cost: a cheap model that consults a strong one only when needed. Anthropic's own advisor tool shows it. On long coding tasks, a cheap executor calling a flagship advisor about once per task hits about 92% of the flagship's score at about 63% of the cost. Real, and it matches our finding. The catch is that it depends on how often you call the expensive model. Rare, on long tasks, is cheap. Constant, on short tasks, is not. That is a per-task routing call. And Anthropic's version routes only among Anthropic models, while the open version routes across all your providers.
The pick is where the real gains hide, and we tested it two ways. First a strong coordinator that grills the candidates, runs their code, and decides picked the correct answer 12 of 12 on the hardest contested tasks, but it was strong enough to solve them itself, so that was only suggestive. Then the clean test: the weakest model in the study, which solves just 2 of those 12 alone, still picked the correct answer 9 times, and 7 of 10 on tasks it could not solve at all. Recognizing the right answer looks easier than producing it, even for a weak model. Honest caveat: this is a 12-task probe, single sample, so we treat it as a strong lead to confirm, not a settled result.
What we are testing next, two hypotheses. First, that recognition result at full scale, many judges, many tasks, repeated samples, to see whether a weak model can reliably pick answers it cannot produce. Second, and the bigger one: this study is single-shot competitive programming, the adversarial case for coordination, so surviving it at all is respectable. The real test is long-horizon agentic work with complementary roles, one model reasons, another generates, a third verifies, over real multi-step loops. If coordination beats the best single model anywhere, it is there. Both ship with the same receipts, and the methods that win become ATO's Pro coordination recipes.
Here is the deal. When you support ATO, you are funding real, open coordination research, run on real keys, with real receipts, and shared with the community. Not a vendor's marketing numbers you cannot reproduce, but experiments you can re-run and build on. Back the tool, fund the research, get the receipts.
Three ways in. All open, all reproducible. 49 tasks is not gospel, and that is the point: run it on your own models.
Download ATO, add your keys, run one line. Any model set you want, no charge. You do the steps.
ato bench run --models "gpt-5,gemini-3-flash,your-model" --suite coordination
Download ATO, free
Log in and watch the actual transcript that produced these numbers, with every receipt. This is the open box Fugu will not show you.
See the receiptsLet ATO run the benchmark on your models on a schedule, and apply the routing pick automatically. The effort is free. The button is Pro.
Get the automationOn the honesty ledger: this is 49 tasks, run once at temperature 0 (with a fourfold resample on the contested ones). That is enough to be directional, not a universal law, which is exactly why the tool ships the suite instead of just the claim. The "44/49 ceiling" is a coverage number, meaning what two models could solve if you always picked right, not a score any recipe reached. Costs: seven of the nine solo rows and every metered recipe ran on per-token API keys, so those costs are receipts, not guesses. Two rows went through the Claude subscription CLI (the sonnet row, and 35 of fable's 49 tasks): grading there is identical, but the CLI does not bill per call, so those cost cells are API-equivalent estimates and are flagged as such in each receipt. Fable's headline cost comes from its 14 metered tasks. Contamination is per-model, not blanket: gemini, gpt-5, and o3-mini rows are contamination-clean (tasks postdate their training cutoffs); the two Claude models have cutoffs after the task window, so they are contamination-exposed; DeepSeek publishes no cutoff, so unknown. The clean-model headline is gemini's 41/49, and no team beat that either. Your keys, your models, your numbers.
Why these four labs. We tested models from OpenAI, Anthropic, Google, and DeepSeek because they are among the best labs shipping right now on public coding and reasoning tests, they include both the leading closed providers and a leading open-weight one (DeepSeek), and together they cover the whole price range a real developer routes across, from a frontier model at about ten cents a task down to a cheap one at a fraction of a cent. We also tried adding z.ai (the GLM models), but it kept draining our API credit far faster than the usage it reported, cents on paper while dollars left the balance. We reached out to z.ai to sort it out and got no response, so we could not get it working for the coordination runs. Since this whole project lives or dies on trustworthy receipts, we stopped testing it rather than publish a cost we could not verify. That is our own experience with our setup, not a verdict on the provider.