// Chapter 02 · Capability

How Close Are Open Models to Closed Ones?

The distance depends on the task. On the tests this report uses, closed models are still ahead: by about two points in graduate-level science and in long agentic tasks, and by 17 to 32 points in advanced mathematics, scientific images and factual knowledge.

Sources: Epoch AI (CC BY 4.0); Vals AI; Artificial Analysis.

2–32points

of difference between the best open-weight model and the best closed model on four tests: 2.2 in graduate-level science, 16.8 in advanced mathematics, 31.8 in factual knowledge and 1.5 in long agentic tasks, on Epoch AI’s and Vals AI’s tests.

Epoch AI (CC BY 4.0); Vals AI.

Illustration: layered landscape with a pale sun
Capability

In graduate-level science, the best open-weight model we can compare, Kimi K3.1 Pro, came within 2.2 points of the best closed one, Gemini 4 Argon. In long agentic tasks it came within 1.5. In advanced mathematics and factual knowledge the distance was 16.8 and 31.8 points. The distance changes from one task to another. But benchmarks are also starting to saturate, and we have changed the way we measure.

01 Four tests, four distances

The distance depends on the task

In graduate-level science, Kimi K3.1 Pro is the best open-weight model, 2.2 points behind Gemini 4 Argon. In factual knowledge, Claude Fable 5.2 is 31.8 points ahead of Kimi K3.1 Pro.

The best open-weight model and the best closed model on four tests

Best open-weight model Best closed model

0%25%50%75%100%Graduate-level scienceKimi K3.1 Pro 94.2%Gemini 4 Argon 96.4%2.2 ptsAgentic work *Kimi K3.1 Pro 63.6%Claude Fable 5.2 and Claude Opus 5.5 65.2%1.5 ptsAdvanced mathematicsKimi K3.1 Pro 78.2%Claude Fable 5.2 95.1%16.8 ptsFactual knowledgeKimi K3.1 Pro 62.2%Claude Fable 5.2 94.0%31.8 pts
  • Graduate-level science

    Kimi K3.1 Pro 94.2%

    Gemini 4 Argon 96.4%

    2.2 pts

  • Agentic work *

    Kimi K3.1 Pro 63.6%

    Claude Fable 5.2 and Claude Opus 5.5 65.2%

    1.5 pts

  • Advanced mathematics

    Kimi K3.1 Pro 78.2%

    Claude Fable 5.2 95.1%

    16.8 pts

  • Factual knowledge

    Kimi K3.1 Pro 62.2%

    Claude Fable 5.2 94.0%

    31.8 pts

Share of each test solved or answered correctly, models released by 10 November 2026, each at its best reasoning setting: Epoch AI’s own runs for three tests. *Agentic work: Vals AI’s Terminal-Bench 4.0, every model through the same agent, mean of three runs. Sources: Epoch AI (CC BY 4.0); Vals AI.
5 more tests from Epoch AI
0%25%50%75%100%Competition mathematics (both at 100%)Kimi K3.1 Pro 100.0%Claude Fable 5.1 and 10 more 100.0%0.0 ptsSoftware engineering (last run in June 2026)GLM-5.2 78.7%Claude Opus 4.7 83.5%4.8 ptsChess puzzlesDeepSeek V4.1 Flash 48.0%Claude Fable 5.2 79.0%31.0 ptsResearch-level mathematicsKimi K3.1 Pro 65.8%Claude Fable 5.2 and GPT-6.1 Sol 100.0%34.2 ptsMystery puzzlesDeepSeek V4.1 Flash 49.0%Claude Fable 5.2 90.0%41.0 pts
  • Competition mathematics (both at 100%)

    Kimi K3.1 Pro 100.0%

    Claude Fable 5.1 and 10 more 100.0%

    0.0 pts

  • Software engineering (last run in June 2026)

    GLM-5.2 78.7%

    Claude Opus 4.7 83.5%

    4.8 pts

  • Chess puzzles

    DeepSeek V4.1 Flash 48.0%

    Claude Fable 5.2 79.0%

    31.0 pts

  • Research-level mathematics

    Kimi K3.1 Pro 65.8%

    Claude Fable 5.2 and GPT-6.1 Sol 100.0%

    34.2 pts

  • Mystery puzzles

    DeepSeek V4.1 Flash 49.0%

    Claude Fable 5.2 90.0%

    41.0 pts

Same reading as the chart: models released by 10 November 2026, each at its best setting, in Epoch AI’s own runs; every other test in Epoch AI’s archive with an open-weight and a closed result. Epoch AI has not run SWE-bench Verified since June 2026, so newer models are missing from it. Source: Epoch AI (CC BY 4.0).

Two of the four distances are small: 2.2 points in science and 1.5 in long agentic work, both for Kimi K3.1 Pro. Two are large: 16.8 in advanced mathematics and 31.8 in answering factual questions from memory, where Claude Fable 5.2 is far ahead of every other model.

02 How far behind in time

How long open models take to reach the closed models’ scores

Take the score the best open-weight model has today on each test, and look back: when did a closed model first reach it? In graduate-level science, Gemini 3.1 Pro Preview did in February 2026, 9 months earlier. Across the four tests, the answer runs from 2 to 11 months.

How many months earlier a closed model reached the score the best open-weight model has today

  1. 1

    Factual knowledge Gemini 3 Flash Preview, December 2025

    11 months

  2. 2

    Graduate-level science Gemini 3.1 Pro Preview, February 2026

    9 months

  3. 3

    Advanced mathematics GPT-5.4 Pro, March 2026

    8 months

  4. 4

    Agentic work Claude Opus 5.5, September 2026

    2 months

For each test: the score of the best open-weight model on 10 November 2026; the first month in which a closed model had reached it, and which model. Models released by each month’s end. Sources: Epoch AI (CC BY 4.0); Vals AI.
The same reading on three more tests from Epoch AI
  1. 1

    Chess puzzles GPT-5 Pro, October 2025

    13 months

  2. 2

    Software engineering Claude Opus 4.6, February 2026

    9 months

  3. 3

    Research-level mathematics GPT-5.5 Pro, April 2026

    7 months

  4. 4

    Mystery puzzles GPT-5.5, April 2026

    7 months

  5. 5

    Competition mathematics Claude Fable 5, June 2026

    5 months

For each test, the first month in which a closed model had reached the score the best open-weight model has today. Source: Epoch AI (CC BY 4.0).

Best score of an open-weight and of a closed model, by month, January 2024 to November 2026

Best open-weight model Best closed model Best open-weight score today First month a closed model reached it

GPQA Diamond · 9 months

0%25%50%75%100%202420252026February 2026a closed model already scored 94.2%
The dashed line is the best open-weight score today; the dot, the first month a closed model reached it. Agentic work: Vals AI ran Terminal-Bench 4.0 on models released in 2026 only, so its lines start late. Sources: Epoch AI (CC BY 4.0); Vals AI.

The time it takes to reach the closed models is shortest where open models are closest: 2 months in agentic work and 9 in science. It is longest in factual knowledge, 11 months, where the gap in points is also the widest.

03 Saturation

In graduate-level science, the leading models are close to the test’s ceiling

In graduate-level science, 30 of the 38 model versions released since July 2026 score above 90%. When almost every leading model is near the top, a difference of 2.2 points is not enough to base decisions on.

On GPQA Diamond the twelve best versions fit within 2.5 points and their standard errors overlap

Score of each version released since July 2026, with its standard error (±1) as Epoch AI publishes it. The axis starts at 85%: this is a dot chart, not a bar chart.

Open-weight Closed

85%90%95%100%Best version of each other open-weight lab, with its place among the 38Gemini 4 ArgonGemini 4 Argon: 96.4% ± 1.296.4%GPT-6 AstraGPT-6 Astra: 95.8% ± 1.495.8%Claude Sonnet 5.5Claude Sonnet 5.5: 95.6% ± 1.495.6%Gemini 3.8 FlashGemini 3.8 Flash: 95.4% ± 1.495.4%GPT-6.1 SolGPT-6.1 Sol: 95.4% ± 1.495.4%Gemini 3.7 FlashGemini 3.7 Flash: 94.8% ± 1.394.8%GPT-6.1 TerraGPT-6.1 Terra: 94.3% ± 1.594.3%GPT-6 SolGPT-6 Sol: 94.3% ± 1.594.3%Kimi K3.1 ProKimi K3.1 Pro: 94.2% ± 1.494.2%Gemini 3.6 FlashGemini 3.6 Flash: 94.1% ± 1.494.1%Grok 4.6Grok 4.6: 94.0% ± 1.494.0%Claude Opus 5Claude Opus 5: 93.9% ± 1.593.9%17. DeepSeek V4.1 Flash17. DeepSeek V4.1 Flash: 93.2% ± 1.693.2%26. GLM-5.326. GLM-5.3: 90.9% ± 1.690.9%30. Mistral Large 4 (open release)30. Mistral Large 4 (open release): 90.2% ± 1.890.2%34. Inkling Small34. Inkling Small: 88.5% ± 1.988.5%
85%90%95%100%Best version of each other open-weight lab, with its place among the 38Gemini 4 ArgonGemini 4 Argon: 96.4% ± 1.296.4%GPT-6 AstraGPT-6 Astra: 95.8% ± 1.495.8%Claude Sonnet 5.5Claude Sonnet 5.5: 95.6% ± 1.495.6%Gemini 3.8 FlashGemini 3.8 Flash: 95.4% ± 1.495.4%GPT-6.1 SolGPT-6.1 Sol: 95.4% ± 1.495.4%Gemini 3.7 FlashGemini 3.7 Flash: 94.8% ± 1.394.8%GPT-6.1 TerraGPT-6.1 Terra: 94.3% ± 1.594.3%GPT-6 SolGPT-6 Sol: 94.3% ± 1.594.3%Kimi K3.1 ProKimi K3.1 Pro: 94.2% ± 1.494.2%Gemini 3.6 FlashGemini 3.6 Flash: 94.1% ± 1.494.1%Grok 4.6Grok 4.6: 94.0% ± 1.494.0%Claude Opus 5Claude Opus 5: 93.9% ± 1.593.9%17. DeepSeek V4.1 Flash17. DeepSeek V4.1 Flash: 93.2% ± 1.693.2%26. GLM-5.326. GLM-5.3: 90.9% ± 1.690.9%30. Mistral Large 4 (open release)30. Mistral Large 4 (open release): 90.2% ± 1.890.2%34. Inkling Small34. Inkling Small: 88.5% ± 1.988.5%
When nearly every leading model scores above 90% and their margins overlap, their order says little. Source: Epoch AI, Benchmarking Hub (CC BY 4.0).

Kimi K3.1 Pro, the best open-weight model, is number 9 among these versions, in a group whose first and twelfth are about three points apart. On this test the ranking says little; the tests that still separate the leaders are the harder ones: advanced mathematics, factual recall, scientific images and long agentic tasks.

04 Scientific images

Reading laboratory images, the best open-weight model is 22 points behind

VIALS asks a model to interpret the images a life-sciences lab works with every day: gel blots, microscopy, flow cytometry, plasmid maps. Artificial Analysis runs every model three times on each of its 161 tasks and counts a task only when all three answers are right. Claude Fable 5.2 did so on 67.1% of the tasks; Kimi K3.1 Pro, the best open-weight model, on 45.3%.

VIALS-AA: share of scientific-image tasks answered correctly in all three attempts

Open-weight Closed

  1. 1

    Claude Fable 5.2 Anthropic · #1 of 17

    67.1% released Nov. 2026

  2. 2

    GPT-6 Astra OpenAI · #2 of 17

    62.7% released Sept. 2026

  3. 3

    Claude Opus 5.5 Anthropic · #3 of 17

    61.5% released Sept. 2026

  4. 4

    GPT-6.1 Terra OpenAI · #4 of 17

    60.2% released Oct. 2026

  5. 5

    Claude Sonnet 5.5 Anthropic · #5 of 17

    59.6% released Sept. 2026

  6. 6

    Kimi K3.1 Pro Moonshot AI · #10 of 17

    45.3% released Oct. 2026

  7. 7

    DeepSeek V4.1 Flash 1017 DeepSeek · #12 of 17

    40.4% released Oct. 2026

  8. 8

    GLM-5.3 Flash Z.ai · #13 of 17

    39.8% released Aug. 2026

161 tasks on 174 laboratory images (blots, microscopy, flow cytometry, plasmid maps, trees, structures). Each model answers every task three times with a Python sandbox and an image viewer; a task counts only if all three answers are right (pass^3). First tab: the five best models and the three best open-weight ones; second: every open-weight model, with its place among all 17. Models released by 10 November 2026. Source: Artificial Analysis, VIALS-AA, as displayed on 11 November 2026; benchmark by Handshake AI.

It is a new test, published on 6 November 2026. It shows a gap the science quiz hides: on GPQA Diamond Kimi K3.1 Pro is 2.2 points from the best closed model; reading the images scientists use, 21.7. Answering once is easier: on the average of the three attempts the two are 11.8 points apart.

05 Agentic work

In long-horizon agentic tasks, the best open-weight model is 1.5 points behind

Terminal-Bench 4.0 gives an agent hours to finish real work in a terminal: build a service, fix a system, train a model. Run by Vals AI with the same agent for every model, Claude Fable 5.2 and Claude Opus 5.5 completed 65.2% of the tasks and Kimi K3.1 Pro, the best open-weight model, 63.6%.

Terminal-Bench 4.0: share of tasks completed, the five best models, the three best open-weight ones and the best of DeepSeek, same agent for all

Open-weight Closed

  1. 1

    Claude Fable 5.2 Anthropic · #1 of 47

    65.2% released Nov. 2026

  2. 2

    Claude Opus 5.5 Anthropic · #2 of 47

    65.2% released Sept. 2026

  3. 3

    Claude Sonnet 5.5 Anthropic · #3 of 47

    64.1% released Sept. 2026

  4. 4

    Kimi K3.1 Pro Moonshot AI · #4 of 47

    63.6% released Oct. 2026

  5. 5

    GPT-6 Astra OpenAI · #5 of 47

    59.6% released Sept. 2026

  6. 6

    GLM-5.3 Z.ai · #16 of 47

    38.9% released Aug. 2026

  7. 7

    GLM-5.3 Flash Z.ai · #18 of 47

    37.4% released Aug. 2026

  8. 8

    DeepSeek V4.1 Flash DeepSeek · #21 of 47

    31.8% released Oct. 2026

Vals AI ran 47 models released by 10 November 2026 through the same agent (mini-SWE-agent) on the 66 tasks of Terminal-Bench 4.0, three times each; the share is the mean of the three runs, with no partial credit. A quarter of the tasks are software engineering; the rest are science, machine learning, operations, hardware, security and media, each with up to eight hours. The 18 best are listed in the Deep Dive. Vals AI ran no open-weight Qwen model; Qwen3.8 Max (#20) is closed. Source: Vals AI, Terminal-Bench 4.0, leaderboard of 9 November 2026.

Kimi K3.1 Pro beats GPT-6 Astra; the next open-weight model, GLM-5.3, completed 38.9%, 25 points lower. On the 18 software tasks alone the gap is 7.4 points, on too few tasks to read on its own. What each run cost and how long it took is in chapter 04.

Chapter 02 · Closing

How to use these results

There is no single distance between open-weight and closed models. We like to think everything comes down to a benchmark score, but it depends a lot on what the model is for. On some tasks the measured gap is small; on others it is still large. Before choosing, check two things: whether the benchmark resembles the real work, and whether the models were evaluated in a comparable way. Then test the candidates on your own tasks.

What the model is for
There is no single distance between open-weight and closed models. We like to think everything comes down to a benchmark score, but it depends a lot on what the model is for.
Distance by task
Kimi K3.1 Pro is about two points from the best closed models in science and in long agentic work; in mathematics, scientific images and factual recall the gap is still 17 to 32 points.
Before choosing
Check two things: whether the benchmark resembles the real work, and whether the models were evaluated in a comparable way. Then test the candidates on your own tasks.