04 Evidence & Methods
The Economics of Open Models
Where each chart’s numbers come from, in one card per chart. The full calculations are in the files named on each card.
The charts
Cost per solved task
- What it shows
- Three models with almost the same success rate: tokens per task, cost per task attempted, tasks completed, cost per solved task and minutes.
- Source
- Vals AI, Terminal-Bench 4.0 leaderboard, updated 9 November 2026 (vals.ai/benchmarks/terminal-bench-4); the page’s metadata declares CC BY 4.0.
- How we calculated it
- Cost per solved task = cost per task attempted divided by the share of tasks completed, failed attempts included, at the serving provider’s prices in each run (mean of three runs). Tokens per task: input and output, cached input included, over the 66 tasks. Every pair of models with different prices per token is also compared on both measures.
- Main limit
- Vals does not publish the price it used for each model; the effective price per token is cost divided by tokens, cache discounts included. The pairs share models, so they are not independent observations.
- Files
analysis/economics/tb4_2026_11/models.csv·analysis/economics/tb4_2026_11/summary.csv
Best value
- What it shows
- Cost per solved task and success of every model released by 10 November 2026; open-weight and closed models with similar success.
- Source
- Vals AI, Terminal-Bench 4.0.
- How we calculated it
- Models ranked by cost per solved task; open-weight/closed pairs within three points of success compared.
- Main limit
- One benchmark of long agentic tasks with one agent; a similar success rate does not mean the same tasks were completed; a model’s cost depends on the provider that served it (column provider). Muse Ember, released on 9 November, ran at the price recorded on its run date, which may not last.
- Files
analysis/economics/tb4_2026_11/models.csv·analysis/economics/tb4_2026_11/pairs.csv
Speed
- What it shows
- Minutes per task against cost per solved task, and the strong models no other beats on both.
- Source
- Vals AI, Terminal-Bench 4.0.
- How we calculated it
- Mean wall-clock time per task, tools and tests included; the frontier joins the models that completed at least half the tasks and are not beaten on both counts by another such model.
- Main limit
- Time depends on the provider that served each model during the run; it is not chat latency or a property of the weights.
- Files
analysis/economics/tb4_2026_11/models.csv
Sources and notes
Show the 12 notes
- 1
analysis/economics/tb4_2026_11/models.csv(research/editorial/vals-tb4/ch04_tb4_2026_11.py): Kimi K3.1 Pro 63.64% completed, cost per test $4.57, cost per solved task $7.18, 49.3 min; Claude Opus 5.5 65.15%, $12.04, $18.48, 63.5 min; GPT-6.1 Sol 55.05%, $1.73, $3.13, 45.2 min. Mean of three runs. - 2Cost per solved task = cost per test / share completed. Cost per test: Vals AI's "Cost / Test", the API spend per task at the serving provider's prices, every call of the agent included. Tokens per task: the run's input and output tokens (cached input included) divided by the 66 tasks (
token_totals,token_tasks_measuredin the leaderboard view). - 3
analysis/economics/tb4_2026_11/summary.csv,pairs_total741,pairs_inverted197: pairs of models that completed at least 10% of the tasks and whose effective price per token (cost per test / tokens per test, cache discounts included) differs. - 4Vals AI, Terminal-Bench 4.0 leaderboard, updated 9 November 2026 (https://www.vals.ai/benchmarks/terminal-bench-4), downloaded 11 November 2026 (
data/raw/vals/terminal-bench-4/2026-11-11/, manifestdata/manifests/vals_terminal_bench_4_2026-11-11.json). avg@3, pass@1 per run, mini-SWE-agent, 8 hours per task, no partial credit. The page's schema.org metadata declares CC BY 4.0; its footer reads "All rights reserved"; written confirmation requested by the owner. Provider per run: columnprovider. - 5Classes as in Chapter 02 (
analysis/gap/p29/classes_2026_11.csv,analysis/gap/p29/openness_new_models.csv); models outside that table classed inresearch/editorial/vals-tb4/tb4_2026_10.py(CLASS_EXTRA), among them Hy4, MiMo V2.6 and MiniMax-M3.1 Flash, not classified. - 6
models.csv,cost_per_solved_usd: Muse Ember 0.55 (released 2026-11-09, provider Meta); models withsolved_pct≥ 50: GPT-6.1 Sol 3.13 … Claude Opus 5 34.74; Kimi K3.1 Pro 7.18. - 7
analysis/economics/tb4_2026_11/pairs.csv: every open-weight/closed pair with success rates within ±3 percentage points (models that completed at least 10%), 28 pairs;matched_open_cheaper18,matched_median_ratio2.58 (closed / open cost per solved task). - 8
models.csv: GLM-5.3 38.89%, $9.37 per test, 31.54 million tokens per test, $24.11 per solved task, provider Zhipu AI; GLM-5.3 Flash 37.37%, $1.22, 35.66 million, $3.26, provider Fireworks AI; Claude Haiku 5.5 36.36%, $3.12, $8.57; Kimi K3 17.17%, $12.48, $72.67. - 9
models.csv: DeepSeek V4.1 Flash (the 17 October release) 31.82%, $0.54, $1.70; Mistral Large 4 27.78%, $5.18, $18.65. - 10
summary.csv,median_minutes_per_test_open70.46,median_minutes_per_test_closed53.58;models.csv,minutes_per_test= Vals AI's duration per test / 60. Frontier: among models withsolved_pct≥ 50, none both cheaper and faster. - 11OpenHands Index (All Hands AI, Apache-2.0), snapshot of 30 June 2026; tables kept in
analysis/economics/oh2026/. Different tasks (SWE-bench Verified), agent (OpenHands) and period. - 12Talk by David Abitbol, who leads AI operations at Revolut, AI Engineer conference, New York, published on YouTube by AI Engineer on 2026-10-29: "our invoice is 1/10 of what we were spending on Anthropic and OpenAI"; GLM-5.3 self-hosted as the default coding model, closed models still used "as exception". Self-reported; no Revolut press release or engineering post; the cost basis, and whether the GPUs are owned or rented, were not disclosed. Notes and copies:
research/editorial/extreme-edit-pass29/ch06-infra/ch06-infra.md, §5b;raw/web/revolut_glm_notes.md.