04 Evidence & Methods

The Economics of Open Models

Where each chart’s numbers come from, in one card per chart. The full calculations are in the files named on each card.

The charts

  1. Cost per solved task

    What it shows
    Three models with almost the same success rate: tokens per task, cost per task attempted, tasks completed, cost per solved task and minutes.
    Source
    Vals AI, Terminal-Bench 4.0 leaderboard, updated 9 November 2026 (vals.ai/benchmarks/terminal-bench-4); the page’s metadata declares CC BY 4.0.
    How we calculated it
    Cost per solved task = cost per task attempted divided by the share of tasks completed, failed attempts included, at the serving provider’s prices in each run (mean of three runs). Tokens per task: input and output, cached input included, over the 66 tasks. Every pair of models with different prices per token is also compared on both measures.
    Main limit
    Vals does not publish the price it used for each model; the effective price per token is cost divided by tokens, cache discounts included. The pairs share models, so they are not independent observations.
    Files
    analysis/economics/tb4_2026_11/models.csv · analysis/economics/tb4_2026_11/summary.csv
  2. Best value

    What it shows
    Cost per solved task and success of every model released by 10 November 2026; open-weight and closed models with similar success.
    Source
    Vals AI, Terminal-Bench 4.0.
    How we calculated it
    Models ranked by cost per solved task; open-weight/closed pairs within three points of success compared.
    Main limit
    One benchmark of long agentic tasks with one agent; a similar success rate does not mean the same tasks were completed; a model’s cost depends on the provider that served it (column provider). Muse Ember, released on 9 November, ran at the price recorded on its run date, which may not last.
    Files
    analysis/economics/tb4_2026_11/models.csv · analysis/economics/tb4_2026_11/pairs.csv
  3. Speed

    What it shows
    Minutes per task against cost per solved task, and the strong models no other beats on both.
    Source
    Vals AI, Terminal-Bench 4.0.
    How we calculated it
    Mean wall-clock time per task, tools and tests included; the frontier joins the models that completed at least half the tasks and are not beaten on both counts by another such model.
    Main limit
    Time depends on the provider that served each model during the run; it is not chat latency or a property of the weights.
    Files
    analysis/economics/tb4_2026_11/models.csv

Sources and notes

Show the 12 notes

  1. 1
    analysis/economics/tb4_2026_11/models.csv (research/editorial/vals-tb4/ch04_tb4_2026_11.py): Kimi K3.1 Pro 63.64% completed, cost per test $4.57, cost per solved task $7.18, 49.3 min; Claude Opus 5.5 65.15%, $12.04, $18.48, 63.5 min; GPT-6.1 Sol 55.05%, $1.73, $3.13, 45.2 min. Mean of three runs.
  2. 2
    Cost per solved task = cost per test / share completed. Cost per test: Vals AI's "Cost / Test", the API spend per task at the serving provider's prices, every call of the agent included. Tokens per task: the run's input and output tokens (cached input included) divided by the 66 tasks (token_totals, token_tasks_measured in the leaderboard view).
  3. 3
    analysis/economics/tb4_2026_11/summary.csv, pairs_total 741, pairs_inverted 197: pairs of models that completed at least 10% of the tasks and whose effective price per token (cost per test / tokens per test, cache discounts included) differs.
  4. 4
    Vals AI, Terminal-Bench 4.0 leaderboard, updated 9 November 2026 (https://www.vals.ai/benchmarks/terminal-bench-4), downloaded 11 November 2026 (data/raw/vals/terminal-bench-4/2026-11-11/, manifest data/manifests/vals_terminal_bench_4_2026-11-11.json). avg@3, pass@1 per run, mini-SWE-agent, 8 hours per task, no partial credit. The page's schema.org metadata declares CC BY 4.0; its footer reads "All rights reserved"; written confirmation requested by the owner. Provider per run: column provider.
  5. 5
    Classes as in Chapter 02 (analysis/gap/p29/classes_2026_11.csv, analysis/gap/p29/openness_new_models.csv); models outside that table classed in research/editorial/vals-tb4/tb4_2026_10.py (CLASS_EXTRA), among them Hy4, MiMo V2.6 and MiniMax-M3.1 Flash, not classified.
  6. 6
    models.csv, cost_per_solved_usd: Muse Ember 0.55 (released 2026-11-09, provider Meta); models with solved_pct ≥ 50: GPT-6.1 Sol 3.13 … Claude Opus 5 34.74; Kimi K3.1 Pro 7.18.
  7. 7
    analysis/economics/tb4_2026_11/pairs.csv: every open-weight/closed pair with success rates within ±3 percentage points (models that completed at least 10%), 28 pairs; matched_open_cheaper 18, matched_median_ratio 2.58 (closed / open cost per solved task).
  8. 8
    models.csv: GLM-5.3 38.89%, $9.37 per test, 31.54 million tokens per test, $24.11 per solved task, provider Zhipu AI; GLM-5.3 Flash 37.37%, $1.22, 35.66 million, $3.26, provider Fireworks AI; Claude Haiku 5.5 36.36%, $3.12, $8.57; Kimi K3 17.17%, $12.48, $72.67.
  9. 9
    models.csv: DeepSeek V4.1 Flash (the 17 October release) 31.82%, $0.54, $1.70; Mistral Large 4 27.78%, $5.18, $18.65.
  10. 10
    summary.csv, median_minutes_per_test_open 70.46, median_minutes_per_test_closed 53.58; models.csv, minutes_per_test = Vals AI's duration per test / 60. Frontier: among models with solved_pct ≥ 50, none both cheaper and faster.
  11. 11
    OpenHands Index (All Hands AI, Apache-2.0), snapshot of 30 June 2026; tables kept in analysis/economics/oh2026/. Different tasks (SWE-bench Verified), agent (OpenHands) and period.
  12. 12
    Talk by David Abitbol, who leads AI operations at Revolut, AI Engineer conference, New York, published on YouTube by AI Engineer on 2026-10-29: "our invoice is 1/10 of what we were spending on Anthropic and OpenAI"; GLM-5.3 self-hosted as the default coding model, closed models still used "as exception". Self-reported; no Revolut press release or engineering post; the cost basis, and whether the GPUs are owned or rented, were not disclosed. Notes and copies: research/editorial/extreme-edit-pass29/ch06-infra/ch06-infra.md, §5b; raw/web/revolut_glm_notes.md.