Which model actually earns its keep?
We run every model through the work Envoy does every day — answering from policy documents, building quizzes, writing reports, converting messy PDFs, calling tools, closing the books. Deterministic grading where the answer can be checked by code; where it can't, three separate AI models mark every answer blind and we publish how much they disagreed. The task set is frozen as v1; new models join the same board as they ship, so scores stay comparable.
Performance by scenario
One eval, five ways to score it: the per-item reference, where every task counts the same, and four weightings that reweight the nine categories to match a real workload. The cards show the strongest results under each — not a verdict on the models — and a shared place (=1) means the scores sit inside the eval's resolution. Click a card to score the whole page that way.
Leaderboard
Category bars, left to right: QA · quiz · report · summary · retrieval · conversion · tools · problem solving · programming. Taller is better; faded bars carry no weight in this scenario.
Tiers follow each maker's own line-up: frontier is its flagship, workhorse its everyday mid-size model, budget its small, cheap one. Click a tier to filter the board.
| # | Model | Weights | Score | Categories | Standout | Run cost | Median latency | Reliability |
|---|
Quality against cost and speed
How much does cost matter to you?
The leaderboard assumes cost is no object and the cheapest-first view assumes quality isn't. Neither is true for a real budget. Drag the slider to say how much cost should count, and the board re-ranks between the two.
Does the winner depend on the weighting?
Each model's best and worst rank across the four scenarios. A short bar is a model that holds its place whatever you ask of it; a long bar is a specialist whose position depends entirely on the job. The filled dot is where it sits under the weighting you have selected.
Every score on the board
Nine categories, every model, darker is better. Hatched cells are n/a — a category the model could not attempt, dropped from its score rather than counted as a failure. Sorted by the weighting you have selected.
swipe the grid sideways for the remaining categories →By category
Top three per category, independent of any scenario weighting. This is the raw material the scenarios are built from.
Open vs proprietary
What a year bought you
A caution on reading the performance line as pure capability: two of the nine categories reward knowing what has happened recently. A model released last year has an older knowledge cutoff and loses ground on the report task for that reason alone, so part of the slope is recency rather than reasoning. The cost line has no such problem — that is a clean measure of what the same quality costs over time.
Checking the graders for bias
The five open-ended items — two quizzes, a report, a summary and a recommendation — are about a third of the composite, and they are the part a model could be flattered on. So they are marked blind by three different models, under shuffled anonymous labels, and the labels are only mapped back after every score is locked. All three graders have a model on the board, and that is the point of the audit: it lets us measure whether a grader is kind to itself or to its own family.
This time it says yes, for two of the three. Allowing for how strict each grader is overall, GPT-6.1 Sol marks its own answers above the other two on a 0–5 scale and Claude Opus 5.5 ; GLM 5.3 sits at . It is the second panel running in which the Sol line leans: the previous panel's GPT-5.6 Sol marked its successor +0.68 above the rest. Averaging three graders keeps any one lean to a third of its weight, and neither changes the top: without its own marks Opus 5.5 still leads, and without Sol's GPT-6.1 Sol moves from fifth to sixth. The full grader-by-model matrix is below for anyone who wants to check; it is an audit of the marking, not a ranking of the models — a model that leads on the checkable categories can sit well down it.
Show the full grader-by-model matrix
Latency
Median per-item response time across the whole run — documents, images and tool calls included, so this is workload latency, not time-to-first-token marketing.
How the eval works
Release dates. Each model shows when it was released, taken from its first appearance in the OpenRouter catalogue. It is there so you can weigh a score against how old the model is — a year-old model holding its own against last month's is a different result from the same score fresh out.
Versioned, not rounds. The 37-item task set is frozen as v1. New models are added to this board as they ship and stay directly comparable with everything already on it. When the tasks themselves change, that becomes v2 — a fresh board, not a reshuffle of this one.
Nine categories, five weightings. Every model gets one score per category. The scenarios are different weightings over the same nine numbers, published so you can check them or make your own. Per item is the derived reference: no editorial judgement at all, every task counting equally, so a category is worth exactly as many items as it holds.
Deterministic first. Everything numeric, exact-match, table F1, the Ruby test suite and forecast RMSE are graded by code. Confabulation traps — questions the source can't answer — are scored on whether the model says so. Code has format assumptions, and they get found: on 5 September 2026 the quiz checker was found to count an answer key's numbered lines as extra questions, to miss options written as Markdown bullets, and to miss explanations given inline after each keyed answer. Fixed, it raised quiz scores on 46 items across 41 models, by up to 0.375 on a single item, and moved 39 composites — all upward, none by more than 0.037. The graders' rubric marks were unaffected, since they read the quizzes rather than parsing them.
Blind where it matters. Open-ended items are reviewed under shuffled anonymous labels by graders who have never seen the answer key, scored 0–5 on fixed rubric dimensions, and only unblinded after scores are locked.
Three graders, and the whole board re-marked on 7 October 2026. Every open-ended item is scored independently by three models — Claude Opus 5.5, GPT-6.1 Sol and GLM 5.3 — and the published rubric score is the mean of the three. They replaced the previous panel (Claude Fable 5, Claude Opus 5, GPT-5.6 Sol and Qwen 3.8 Max), and rather than mix the two, the new panel re-marked every model on the board. Opus and Sol agree at , Opus and GLM at , Sol and GLM at . Leave any one grader out and Claude Opus 5.5 still leads; roughly half the board shifts, most often by a single place, and the average score changes by less than 0.006. Which model marks the work is not what decides the ranking.
We checked the graders for self-interest, and two of them lean. All three graders have a model on this board, so we measured whether a grader scores its own model's answers above what the other two give them. Allowing for how strict each grader is overall, GPT-6.1 Sol marks its own answers higher on a 0–5 scale and Claude Opus 5.5 ; GLM 5.3 is close to neutral at . Opus and Sol also lean towards their own company's other models ( and ); GLM does not (). The table in the graders section has every figure. Each figure rests on five items, so treat it as noisy — but it is the second panel running in which the Sol line leans: the previous panel's GPT-5.6 Sol marked its successor GPT-6 Sol +0.68 above the others. Averaging three graders cuts any one leaning to a third of its weight, and we checked the cases that matter: with Opus 5.5's own marking removed it still leads; with Sol's removed, GPT-6.1 Sol moves from fifth to sixth.
One grader marks through the API, not the repository. GLM 5.3 reads each pack as API requests with no access to this repository, so it could not reach the answer key or the label map even had it tried — the blinding is structural rather than promised. It is the most generous of the three, marking about a quarter of a point above the panel on average, which lifts every model alike. It reads the main pack in groups of twenty candidates against the same rubric, with its reasoning capped at medium effort so it has room left to reply; a grader comparing twenty answers at a time sees a narrower field than one reading all seventy-one. It was vetted before joining: it agrees with Opus as closely as Opus agrees with Sol, and its scores show no drift with label position.
Honest n/a. Text-only models aren't punished for vision items; endpoints without function calling aren't punished for tool items. Those categories are dropped and the score reweighted — refusals and errors, though, score zero.
Eight models were answered in agent sessions, not through the API. Claude Opus 5.5, Claude Fable 5.1 and Claude Sonnet 5.5 (Claude Code), and GPT-6 Astra, GPT-6.1 Sol and GPT-6 Luna (Codex), answered all 37 items in a session on 7 October 2026, Claude Haiku 5.5 (Claude Code) on 8 October — Fable because no API route for it is open under this account's zero-data-retention setting, the rest by choice — and GPT-6 Sol in a Codex session in September, because its API route was rate-limited all launch day. Each session was given exactly what an API run gets: the same system prompt and user message, the same attached documents, data inlined where the API path inlines it, and the four tool functions through a logged call shim. It was barred from web search, code execution, any file outside its pack, and revising an earlier answer. What differs is the wrapper — an agent harness instead of a bare API call, no fixed temperature or token cap, no metering and no timing — so those rows carry a session label. Six of the eight replace a predecessor that ran over the API, so their step up from it mixes the model with the method.
Run cost is at today's list price. Every model's run cost is its recorded token use on all 37 items, priced at the model's current list price: the maker's own standard API rate where it serves the model, otherwise the median across providers, excluding discounted flex and batch tiers and premium fast or priority ones. What a run was actually billed depended on when it ran and which provider OpenRouter routed it to — GPT-5.6 Sol was billed at three times today's price in August, before a price cut — so billed costs would compare runs from different months, not models. One price is temporary: Gemini 4 Argon is priced at Google's launch rate of $2 in and $10 out per million tokens, which doubles to $4 and $20 on 1 December 2026; its run cost will roughly double with it. The money actually spent running the eval is in the totals.
Run cost and latency for session runs. A session is not metered or timed, so seven of the eight were also run once through the API, and their run cost and latency come from that run. Fable 5.1 has no open API route here, so its figures come from Fable 5's token use priced at Fable 5.1's list price, and Fable 5's response time — treat those two as indicative.
Both runs count. For the seven models with a session run and an API run, each auto-marked item scores the average of the two; the written items keep the judges' marks of the session answers. The API runs scored between 0.03 below and 0.05 above their sessions, so averaging halves the luck of any one answer.
What changed, and when — the board's history in order
The published weighting follows discrimination, and changed on 5 August 2026. Weight now sits where models actually differ. Grounded QA has twelve items but 53 of 56 models score a perfect 1.00 on it — it is a gate everyone passes, and weighting it heavily measured very little. Programming, problem solving and report writing spread the field by 0.63 to 0.75 and were carrying less weight than they revealed. So qa fell from 18% to 10% and retrieval from 14% to 10%, while problem solving rose to 16% and programming to 14%. The board separates models better as a result — the spread across the field widened from 0.33 to 0.40.
What that change did, stated plainly. It moved 46 of 56 models, and it moved Claude Fable 5 from third to first. We are telling you that because a weighting revised after seeing results, in a direction that happens to restore a previous leader, is exactly the thing you should be suspicious of. The rule was fixed before the board was recomputed, it is stated above, and the per-item view is published alongside so you can see the board with no editorial weighting at all. It also made the eval less precise per item, not more: with weight on categories holding fewer tasks, dropping any single item now moves a score by ±0.019 rather than ±0.014.
Re-runs, and why. The first pass had two faults of our own making. A token cap set too low let reasoning models spend their whole budget thinking and return nothing visible, which scored zero on 30 items across 15 models; and a provider guardrail truncated some responses mid-sentence. Both are properties of the harness and the route, not of the models, so the affected items were re-run with a higher cap and the scores below reflect the second attempt. Every re-run item is counted in the table, and the raw first-pass results are kept.
Where a route failed entirely. A handful of items kept returning truncated output on every provider that serves that model. Rather than score a model zero for its router, those answers were produced through a direct session. They are not like-for-like with the provider runs — different plumbing, no document attachment path — and the models carrying them are listed below.
Three grading bugs, fixed on 24 September 2026. A quiz that numbers its questions as Markdown headings (### 1.) was counted as having none; bold option letters (**A.**) were not recognised as options; and a LaTeX thousands separator (12{,}500) was read as two numbers, so a correct, boxed break-even was marked missing. All three were fixed for every model, and every one that moved, moved up. A score can be wrong when the run is perfect, so a surprising drop now gets its item-level detail read before it is published.
How the 7 October board was marked. Seventy-one models were reviewed in one blind pack. Four more — Mercury 2.5, GPT-6 Sol, Qwen 3.8 Flash and Grok 4.6, whose answers were produced on a second machine — were reviewed afterwards as their own pack, mixed blind with ten anchor candidates the panel had already scored, chosen to span the board from GPT-OSS 120B to Claude Opus 5. Each grader ranked the anchors almost exactly as before (r = 0.98–0.99); on level, Opus moved −0.03, GLM −0.10 and Sol +0.28, marking that small pack more generously. The four are published as marked, not corrected: the shift is worth about 0.004 of a composite, inside the eval's resolution. The anchors keep their main-pack scores. The September additions had been graded the same way by the previous panel in two anchored packs; those marks are archived, not used. Claude Haiku 5.5, added on 8 October, was marked the same way in its own pack with ten anchors (r = 0.82–0.91; Opus +0.01, GLM −0.04, Sol +0.21), published as marked. Gemini 4 Argon, added on 14 October and run through the API, had its own anchored pack too (r = 0.83–0.91; Opus −0.07, GLM −0.12, Sol +0.26). Sol's shift is above the quarter-point we flag; across the panel the shift is +0.02, worth about 0.001 of Argon's composite, so it is published as marked. Leave Sol's marks out and Argon stays fourth. Muse Wonder, added the same day and also run through the API, was marked in a pack of its own (r = 0.87–0.92; Opus −0.06, GLM −0.13, Sol +0.14; panel −0.02), published as marked. Its anchors included Argon, scored in the pack before.
Five more grading faults, fixed on 7 October 2026. Reading every new answer turned up five more faults in our own grading. Each marked correct work wrong, and each fix was applied to every model on the board. Confabulation traps missed plain refusals worded “does not give” or “couldn’t find”, or typed with a curly apostrophe — and let two invented provider counts through because of a non-breaking hyphen. The quiz count still missed questions headed “Question 1 (Multiple choice)”, and numbered solution steps could stand in for missing questions. The two-column conversion compared reading order against a text extraction that interleaved the columns, so models that read the page correctly were marked down. The scanned-page conversion rewarded a heading the page does not have and never checked its two columns; it now scores reading order, the pull-quote and the footnotes. And the summary pack given to graders had never included the report being summarised, so fidelity had been judged without the source on every board until now — it now does, and every summary was re-marked against it.
When a model said nothing, we found out why. Three items came back empty: the request succeeded, the whole completion budget went on hidden reasoning, and no answer appeared. Raising the budget did not help — it only bought more thinking. Capping the reasoning instead, so the model had room left to reply, produced an answer on all three within a few thousand tokens. Those caps are recorded per model in the registry and applied to every item that model runs, not just the one that failed. It is worth knowing that a reasoning model can be unable to answer at any budget and answer easily at a smaller one. The September additions needed the same treatment — six empty items across GLM 5.3, GLM 5.3 Flash and Qwen 3.8 Flash, all answered once reasoning was capped, and two of those routes ignored an effort level and honoured only a token budget. One took a fifth attempt: GLM 5.3 on the inline forecast item returned nothing at 32,000 tokens on four attempts across three providers, all of which ignored the cap. Reasoning cannot be switched off on that model at all — the endpoints reject the request — but a fourth provider honours the budget and the model answered in under 5,000 tokens. That route is recorded on the item and the answer is scored like any other. Two models never answered at any setting tried and are off the board rather than carrying zeros for silence: MiMo v2.6 Pro, whose inline forecast came back empty on six attempts across two machines and three providers, and Command A+, whose provider ignored a reasoning budget and which left four to nine items empty on each of four passes. GLM 5.3 and GLM 5.3 Flash are published from 7 October runs capped at medium effort, on which every item answered.
Six more grading faults, and session-run costs, 8 October 2026. Reading the board after publishing turned up six more format assumptions, each fixed for every model: explanations given on a bolded key line (**4. B** — …) or on a “Correct answer: B — …” line were not recognised; quiz options laid out inside a Markdown table were not counted; “4th largest” was not read as “fourth”; and a summary's word count included its Markdown symbols, pushing five summaries over the 400-word limit they were inside. Twelve answers moved, all upward. The same day, every run cost moved to today's list price (see above), the session runs were given run costs and latencies instead of a dash, and Mistral Large 4 was marked open, weights due 27 October 2026. A refusal worded “the document doesn’t say” is now recognised too; no published answer was worded that way.