02 Evidence & Methods

How Close Are Open Models to Closed Ones?

Where each chart’s numbers come from, in one card per chart. The full calculations are in the files named on each card.

The charts

  1. Four tests, four distances

    What it shows
    The best open-weight model and the best closed model on four tests, named, and the distance between them in points.
    Source
    Epoch AI’s benchmark runs (CC BY 4.0), archive of 10 November 2026; Vals AI, Terminal-Bench 4.0, leaderboard of 9 November 2026; our verification of public weights.
    How we calculated it
    For each test, the best run of each model version released by 10 November 2026; the distance is best closed minus best open-weight. Open-weight from the day the weights were public (Kimi K3.1 Pro: 4 November). Gated weights (Muse Spark Open) and releases our registry has not classified (Hy4, MiMo-V2.6) are not counted as open.
    Main limit
    Three tests in Epoch’s own harness; the agentic test in Vals AI’s, with mini-SWE-agent for every model. SWE-bench Verified, used in earlier editions, is no longer run: Epoch AI rated it flawed on 3 September 2026 and its last runs date from June. Earlier versions of this chapter used a cut-off of 31 October 2026.
    Files
    analysis/gap/p29/frontier_2026_11.csv · analysis/gap/p29/tb4_frontier_2026_11.csv · analysis/gap/p29/openness_new_models.csv
  2. How far behind in time

    What it shows
    For each test, the first month in which a closed model reached today’s best open-weight score; the two frontiers by month.
    Source
    Epoch AI’s benchmark runs (CC BY 4.0); Vals AI, Terminal-Bench 4.0.
    How we calculated it
    Monthly frontiers among versions released by each month’s end, the last point on 10 November 2026; the lag is the number of months from that first month to November 2026.
    Main limit
    An observation of four tests, not a rule for how models progress; the frontier moves only when a new release is evaluated. Vals AI ran Terminal-Bench 4.0 on models released in 2026 only.
    Files
    analysis/gap/p29/frontier_monthly_2026_11.csv · analysis/gap/p29/tb4_monthly_2026_11.csv
  3. Saturation

    What it shows
    The twelve best model versions released from July to 10 November 2026 on GPQA Diamond, named, with their standard errors.
    Source
    Epoch AI’s benchmark runs (CC BY 4.0), archive of 10 November 2026.
    How we calculated it
    Each version’s best run; classes as in the first chart.
    Main limit
    Standard errors are Epoch’s, from the sampling of questions; they do not cover repeated runs or changes in the harness.
    Files
    analysis/gap/p29/gpqa_top_2026_11.csv · analysis/gap/p29/saturation_2026_11.csv
  4. Scientific images on VIALS-AA

    What it shows
    The share of 161 tasks on laboratory images that each model answered correctly in all three attempts.
    Source
    Artificial Analysis, VIALS-AA, as displayed on 11 November 2026 (artificialanalysis.ai/evaluations/vials-aa); VIALS benchmark by Handshake AI.
    How we calculated it
    pass^3: a task counts only if all three answers are right; Python sandbox and image viewer for every model; answers graded by a language model with the benchmark’s own prompts. Models released by 10 November 2026, classes as in the first chart.
    Main limit
    A different evaluator from the four tests, so it is not added to them. Artificial Analysis’ terms allow personal, non-commercial use; the owner has requested permission to publish these scores, and this preview shows them while the answer is pending.
    Files
    analysis/gap/p29/vials_2026_11.csv
  5. Agentic work on Terminal-Bench 4.0

    What it shows
    The share of 66 long tasks completed in a terminal by each model, every model through the same agent.
    Source
    Vals AI, Terminal-Bench 4.0 leaderboard, updated 9 November 2026 (vals.ai/benchmarks/terminal-bench-4); the page’s metadata declares CC BY 4.0.
    How we calculated it
    Mean of three full runs per model (avg@3), one attempt per task, no partial credit, up to eight hours per task; mini-SWE-agent for every model. Models released by 10 November 2026, open-weight from the day their weights were public.
    Main limit
    A quarter of the tasks are software engineering; the rest are science, machine learning, operations, hardware, security and media. The 18 software tasks alone give a gap of 7.4 points, on too few tasks to rely on. Earlier agentic benchmarks are not comparable: SWE-bench Verified is saturated and rated flawed by Epoch AI, and DeepSWE 1.1 was rated flawed on 7 September 2026.
    Files
    analysis/gap/p29/tb4_models_2026_11.csv · analysis/gap/p29/tb4_frontier_2026_11.csv

More data

Figures from the research that the chapter’s charts do not show.

  • Claude Fable 5.2, released on 9 November 2026, scored 94.0% on SimpleQA Verified, 18 points above the previous best, Gemini 4 Argon’s 76.0%; without it the distance is 13.8 points. The questions are public and the result has not yet been reproduced, so it calls for caution.
  • On the 18 software tasks of the Vals AI test alone, GPT-6 Astra completed 61.1% and Kimi K3.1 Pro 53.7%: a distance of 7.4 points, on too few tasks to read on its own.

Sources and notes

Show the 31 notes

Evidence levels follow the project's scale: E1 = published by the source and read in the original; E2 = recomputed by us and matching a published figure; E3 = computed by us from the source data (reproducible pipeline and tests). Benchmark data: Epoch AI, Capabilities & benchmarking (epoch.ai/benchmarks), benchmark_data.zip retrieved 2026-10-15, CC BY 4.0, Epoch's own runs only (_external files excluded); manifest data/manifests/epoch_benchmarks_2026-10-15.json. Human preference data: lmarena-ai/leaderboard-dataset, revision a64ca3b, CC BY 4.0; manifest data/manifests/lmarena_2026-10-19.json. Openness classes: Helmcode registry as returned by pipelines.identity.effective.registry() (open side only). All distances are Scenario A of DEC-088: *distance between the best model with verified public weights and the best evaluated model of any class; not a comparison with confirmed closed-weight models. Every number is listed with its source in research/editorial/sprint25/ch02-claims.csv.

  1. 1
    analysis/gap/p29/frontier_2026_11.csv (research/editorial/extreme-edit-pass29/ch02-capability/frontier_2026_11.py): Epoch AI's archive of 10 November 2026 (CC BY 4.0, manifest data/manifests/epoch_benchmarks_post_cutoff_2026-11-10.json), models released by 10 November 2026 (owner decision 2026-11-11; earlier versions used 31 October), one best run per version; open-weight decisions for new releases in analysis/gap/p29/openness_new_models.csv (pending human review). Kimi K3.1 Pro: open from 2026-11-04, the first archived public capture of moonshotai/Kimi-K3.1-Pro (Hugging Face createdAt 2026-11-02). Muse Spark Open's weights are gated, so it is not counted. With the earlier cut-off: frontier_2026_10.csv.
  2. 2
    The three questions (distance in points, time to reach a score, usefulness in production) and the decision to measure only the first two: research/editorial/chapter02-argument.md, §1. Comparison defined in DEC-081; closed-weight decisions suspended in DEC-099 (registry/decision_suspensions.csv); scenario B not computable (analysis/gap/scenarios.csv, computable = False).
  3. 3
    analysis/gap/p29/tb4_frontier_2026_11.csv and tb4_models_2026_11.csv, built by research/editorial/vals-tb4/tb4_2026_10.py from Vals AI's Terminal-Bench 4.0 leaderboard, updated 9 November 2026 (https://www.vals.ai/benchmarks/terminal-bench-4), downloaded 11 November 2026 (data/raw/vals/terminal-bench-4/2026-11-11/, manifest data/manifests/vals_terminal_bench_4_2026-11-11.json). 66 tasks, mean of three runs (avg@3), pass@1 per run, mini-SWE-agent for every model, 8-hour limit per task. The page's schema.org metadata declares CC BY 4.0; its footer reads "All rights reserved"; written confirmation requested. Files tb4_*_2026_11.csv, cut-off 10 November 2026 (TB4_CUTOFF=2026-11-10); classes as in note 29; Hy4 is not classified in our registry and Muse Ember was released on 9 November as a closed model. Gap 1.5 points, standard error 2.9 (per-run standard errors 1.75 and 2.31). Lag: Claude Opus 5.5, released 22 September 2026, scored 65.2%. With the earlier cut-off (tb4_*_2026_10.csv): Claude Opus 5.5 65.2% against GLM-5.3 38.9%, 26.3 points. Software tasks: software_only_* columns. SWE-bench Verified: Epoch AI review of 3 September 2026 (https://epoch.ai/benchmarks/swe-bench-verified/review), Vals AI archive note of 1 September 2026 (https://www.vals.ai/benchmarks/swebench). DeepSWE: Epoch AI review of 7 September 2026 (https://epoch.ai/benchmarks/deepswe/review); results in analysis/gap/p29/coding_2026_10.csv, cited, not redistributed.
  4. 4
    Method: DEC-082; pipelines/gap/runs.py, pipelines/gap/frontier.py; research/gap/candidate-findings.md. Best run across evaluated reasoning efforts per version, release date as the date that counts, open versions from the later of release and the documented weights date (open_valid_from). No normalisation or index across benchmarks (DEC-030, DEC-082).
  5. 5
    OTIS Mock AIME 2024-2025, month ending 2025-07-31: leader of any class and best verified open-weight both Qwen3-235B-A22B-Thinking-2507 (leader_all_evidence = verified_open in frontier_leaders.csv; gap_pp = 0.0 in frontier_monthly.csv); registry class OW-P. LM Arena text / overall, snapshots 2025-01-24 to 2025-02-27 (11 snapshots): top_any = deepseek-r1, gap_any_vs_verified_open_points = 0.0 (analysis/gap/arena_snapshots.csv); registry class of deepseek/deepseek-r1: OW-P. September 2026 leaders: leader_all_evidence = unknown, Epoch labels "API access" or "Limited access" (frontier_leaders.csv). No unverified Epoch-labelled-open candidate above the verified open frontier at 2026-09-30: research/gap/frontier-impact.md, §2.1; DEC-085. Outside Epoch's open labels: qwen3.8-max_xhigh (Epoch "API access", released 2026-08-02, run 2026-08-04) scores 0.75 on FrontierMath Tiers 1–3 v2 against kimi-k3_max 0.72 (analysis/gap/runs.csv); sources/model_reference_2026.csv records Qwen3.8-2.4T-A95B as public on Hugging Face and "hosted as Qwen3.8-Max (API)"; 93.68 − 74.74 = 18.9 points. Pass 26, independent review C-02. E3.
  6. 6
    Classes and evidence rules: research/openness-taxonomy.md, research/openness-evidence-audit.md (DEC-022, DEC-023). Effective classes: pipelines.identity.effective.registry(): moonshotai/kimi-k3-20260715 OW-N, deepseek/deepseek-v4-pro-20260813 OW-P, z-ai/glm-5.2-20260616 OW-P, xiaomi/mimo-v2.6-pro-20260921 OW-N; Meta Llama versions UNK (pending_human). Nine CW decisions of DEC-093 and DEC-094 suspended, effective class UNK (registry/decision_suspensions.csv, DEC-099); confirmed_closed_versions = 0 on every benchmark (analysis/gap/gap_summary.csv). What a closed-side comparison would require: DEC-083 (queue A, analysis/gap/review_queue.csv) and research/gap/scenarios-and-gates.md (gate G2).
  7. 7
    analysis/gap/stat_audit.csv and research/gap/statistical-audit.md (intervals cover question sampling only; winner's-curse diagnostic below 1 point net on every benchmark). analysis/gap/comparability.csv, GPQA Diamond median_delay_days: verified open 125.5, Epoch "API access" label 29.0. Provider: Florian Brand and Jean-Stanislas Denain, "Why benchmarking is hard", Epoch AI Gradient Updates, 2025-12-23, section "Model Provider" (epoch.ai/gradient-updates/why-benchmarking-is-hard, read 2026-11-02; Gradient Updates posts represent the authors' views, not necessarily Epoch's; the test used GPQA Diamond and 2025 open models); research/gap/methodology-audit.md, §1 and §3; 7 runs with a fireworks/ provider prefix in analysis/gap/runs.csv. Gate G6 not met (research/gap/scenarios-and-gates.md). E3. DEC-087, DEC-088.
  8. 8
    research/gap/methodology-audit.md, §2: Haas et al., arXiv 2509.07968 (paper CC BY 4.0; dataset MIT), 1,000 questions filtered from OpenAI's SimpleQA; one turn, no tools; Epoch reports the proportion correct, abstention counted as incorrect, so its scores are not comparable with the paper's F1; grader gpt-5.6-terra; label defects in 10% of a 50-question audit reported in Epoch's review; prompt change on 2026-08-27. Re-read in the original on 2026-11-02: Epoch AI, "SimpleQA Verified" benchmark review (reviewed 2026-09-10), Interpretation, Elicitation & Scaffolding and Limitations, epoch.ai/benchmarks/simple-qa-verified/review; Epoch's benchmark page, Methodology and Changelog, epoch.ai/benchmarks/simpleqa-verified (E1). Sensitivities: analysis/gap/sensitivity_checks.csv ("only runs started on/after 2026-08-27": 23.1; "exclude Epoch 'Limited access' / pre-release models": 22.7, leader gpt-6-astra_max). Interval: analysis/gap/stat_audit.csv and gap_summary.csv (20.6–26.6; the Sprint 13 re-run gives 20.7–26.5, Monte Carlo error). E3. DEC-087.
  9. 9
    analysis/gap/scenarios.csv (Scenario A): gpt-6-astra_max = gpt-6.1-sol_max 0.94 vs kimi-k3_max 0.72 (OW-N), 21.5 points, 17.6–26.3. stat_audit.csv: 83 versions released by 2026-09-30, 13 verified open (gap_summary.csv counts 84, every version in the archive); coverage.csv: benchmark release 2026-06-12. Benchmark conditions, re-read in the original on 2026-11-02: Epoch AI, "FrontierMath Tiers 1-3 (v2)", epoch.ai/benchmarks/frontiermath-tiers-1-3-v2 (introduction: v2 released 2026-06-12, 295 problems in Tiers 1–3, ten public, hub scores on the private sets; Methodology: stateless Python tool) and Epoch's linked implementation (gist.github.com/tadamcz/7b3a45d0d5d35f758bb6deb8000731bb: network_mode: none, automatic scorer). E3 (distance), E1 (conditions).
  10. 10
    OpenAI's access: Epoch AI, FrontierMath "About" page, "Conflict of interest statement" (epoch.ai/frontiermath/tiers-1-4/about, read 2026-11-02): 300 core problems commissioned by OpenAI, with access to statements and solutions except 53 solutions withheld in version FrontierMath_02-28-25; nothing stated for v2. The 50-problem holdout announced on 2025-01-23 (epoch.ai/blog/openai-and-frontiermath) was then still being finalised. E1. analysis/gap/sensitivity_checks.csv: "exclude OpenAI models", leader claude-opus-5-5_max 0.91, distance 19.0; "exclude runs evaluated before the model's release date", 21.5. gpt-6-astra_max evaluated 2026-08-30, released 2026-09-03; gpt-6.1-sol_max evaluated and released 2026-09-29 (research/gap/frontier-impact.md, §1). E3. DEC-087; gate G5.
  11. 11
    analysis/gap/frontier_monthly.csv, FrontierMath-Tiers-1-3-v2-Private: first month with a verified open-weight version 2026-02-28 (verified_open_available 1, distance 42.8); 51.2 (2026-03-31), 16.9 (2026-07-31), 21.5 (2026-09-30). coverage.csv: first_eval 2026-06-09, three days before the benchmark's publication on 2026-06-12 (note 8): Epoch, which runs the benchmark, evaluated models before publishing it. E3.
  12. 12
    analysis/gap/frontier_monthly.csv, SimpleQA Verified: gap_pp 9.0 (2025-07-31), 33.1 (2026-02-28 and 2026-03-31), 23.1 (2026-09-30). Evaluation dates: analysis/gap/coverage.csv, first_eval 2026-08-10, last_eval 2026-10-13. E3. Leader rotation (section A Snapshot, Not a Trend): analysis/gap/frontier_leaders.csv, columns leader_open and leader_open_release, months 2026-01 to 2026-09: FrontierMath-Tiers-1-3-v2-Private qwen3.5-397b-a17b_none (2026-02, 2026-03), kimi-k2.6 (2026-04, 2026-05), glm-5.2_max (2026-06), kimi-k3_max (2026-07 to 2026-09); GPQA diamond glm-4.7, glm-5, deepseek-v4-pro_high (released 2026-04-24), glm-5.2_max, kimi-k3_max; SimpleQA Verified Qwen3-235B-A22B-Thinking-2507, deepseek-v4-pro_max, kimi-k3_max, deepseek-v4-pro-0813_max. Same rules as the rest of the chapter (verified open-weight class, valid from its first public capture). E3.
  13. 13
    analysis/gap/ceiling.csv (42 versions above 0.9; benchmark release 2023-11-20; Epoch ceiling 1.0); 238 versions released by 2026-09-30 (analysis/gap/stat_audit.csv, versions). frontier_monthly.csv: 12.1 points (2025-03-31), 1.5 points (2026-07-31), 3.3 points (2026-09-30), interval 1.2–5.6. Leader changes in 74.3% of 2,000 draws (stat_audit.csv, share_draws_leader_all_changes 0.74). Disputed questions: research/gap/candidate-findings.md, C3 (pending quantification). E3. DEC-082 (saturated benchmarks used only to show the ceiling effect).
  14. 14
    analysis/gap/gap_summary.csv (gap_pp_2026-09: OTIS 1.4 points, interval 0.0–1.8; MATH Level 5 1.5 points) and analysis/gap/ceiling.csv (OTIS best_score 1.0; 58 versions above 0.9 in the whole archive, 55 among versions released by 2026-09-30, best run per version in analysis/gap/runs.csv, as for GPQA Diamond's 42 of 238). E3. DEC-082; gate G8 in research/gap/scenarios-and-gates.md.
  15. 15
    analysis/gap/p29/vials_2026_11.csv, from Artificial Analysis' VIALS-AA page as displayed on 11 November 2026 (data/raw/aa_vials/2026-11-11/, manifest data/manifests/aa_vials_2026-11-11.json): pass^3 (all three repeats correct), average of 3 and pass@3; classes by the report's rules, not the source's labels. Artificial Analysis' terms allow personal, non-commercial use; the owner requested permission on 11 November 2026 and the preview shows the scores while the answer is pending.
  16. 16
    analysis/gap/gap_summary.csv, lag_months_2026-09: FrontierMath Tiers 1–3 v2 9, SimpleQA Verified 9, SWE-bench Verified 7, GPQA Diamond 7, MATH Level 5 17 (distance 1.5 points). The SWE-bench 7 depends on the repeated-run rule (note 15). Lag = months since the overall frontier first reached the current open-frontier score (DEC-082). Not averaged or published as a single figure (research/gap/candidate-findings.md, C5). Epoch AI, "Open models lag state-of-the-art closed models by 4 months" (J. Edwards and L. Emberson, 29 May 2026, epoch.ai/data-insights/open-closed-eci-gap, read 2026-11-07): "Since January 2026, the most capable open-weight models have lagged frontier closed models by an average of four months in the Epoch Capabilities Index (ECI)". E3.
  17. 17
    analysis/gap/arena_snapshots.csv, 2026-10-15: "Gemini 4 Argon (High)" 1532.3, mimo-v2.6-pro (OW-N), 41.7 points, expected win probability 0.56, intervals separated; 2026-10-13: 43.0. Without Gemini 4 Argon: 25.1 (research/gap/frontier-impact.md, §2.2; research/gap/scenarios-and-gates.md, Scenario A). Name change and gate G9: frontier-impact.md, §2.3; DEC-085. E3.
  18. 18
    analysis/gap/arena_snapshots.csv, our half-year aggregation of gap_any_vs_verified_open_points and ci_separated_any_vs_verified_open (scenario A). Table text, overall: 2025-S1 median 37.4 (range 0.0–54.0; deepseek-r1 top-rated in 11 snapshots), separated 29 of 42; 2025-S2 43.9, 2026-S1 41.6, 2026-S2 40.6 (27.1–46.6), all separated. The top open-labelled model differs from the top verified one in 14 of 54 snapshots in 2026-S1 and 15 of 29 in 2026-S2, mostly because Arena names such as kimi-k3-max, kimi-k2.5-thinking and glm-5.3-max are not matched to the registry (gate G9). 2024, not reported: no verified open-weight model before 2024-04-18; medians 137.3 (19 of 34 snapshots) and 110.6, overstated because Llama 3 and 3.1, Qwen1.5 and Command R+ have no verified class. Sensitivity with Arena's labels (scenario C, not verified by Helmcode; gap_prop_vs_open_label_points, where "open" is any licence label other than "Proprietary", non-commercial ones included): 2024-S1 79.4, 2024-S2 57.7, 2025-S1 37.4, 2025-S2 43.9, 2026-S1 38.25, 2026-S2 33.9. Half-years pool snapshots across the method changes of 2025-05-16, 2025-07-23 and 2026-05-12. 2026-S2 by category: coding 35.6 (separated 20 of 29), hard prompts 40.5 (29 of 29), math 65.6 (22 of 29); text_style_control overall 38.7. With Arena's labels: 24.1, 36.1, 60.8 and 22.7. Through 2026-10-15. E3. DEC-084, DEC-088.
  19. 19
    Epoch's own runs start 2025-01-27 (coverage.csv, GPQA Diamond first_eval). 36 Epoch-labelled open versions above the verified open frontier (analysis/gap/review_queue.csv, reason "Epoch-labelled open weights above the verified open frontier"; DEC-083, queue B). GPQA Diamond gap_pp_2024-06 38.9 (gap_summary.csv) treated as a coverage artefact: research/gap/candidate-findings.md, C6; DEC-082. Survivorship: candidate-findings.md, "Comparabilidad". LM Arena in 2024: note 21. E3.
  20. 20
    Chapter 03, "The 90-Day Window" (report/chapters/03-follow-the-tokens.md); Chapter 04 on cost per solved task (report/chapters/04-economics-of-open-models.md). Leader changes: analysis/gap/frontier_leaders.csv.
  21. 21
    analysis/gap/frontier_leaders.csv and analysis/gap/scenarios.csv (Scenario A), month ending 2026-09-30. GPQA Diamond: kimi-k3_max 0.93 (OW-N) vs gemini-4-argon_high 0.96; SimpleQA Verified: deepseek-v4-pro-0813_max 0.53 (OW-P) vs gemini-4-argon_high 0.76. Distances gap_pp_2026-09 in analysis/gap/gap_summary.csv. E3. DEC-081, DEC-082. Runs behind each distance (table after Figure 2.1): pipelines/gap/leaders_table.py, analysis/gap/benchmark_leaders_2026_09.csv, from analysis/gap/runs.csv (started_at, reasoning_effort, score, stderr); each distance reproduces gap_pp_2026-09. Best OW-P where the open leader is OW-N: deepseek-v4.1-flash_max 0.67 (FrontierMath Tiers 1–3 v2), glm-5.2_max 0.92 (GPQA Diamond). E3.
  22. 22
    analysis/gap/gap_summary.csv: FrontierMath-2025-02-28-Private 13.4 (9.6–20.4), Epoch ceiling 0.57 (ceiling.csv; superseded_by in coverage.csv); Chess Puzzles 26.0 (17.0–34.3). Versions of a benchmark are not chained (DEC-082). E3.
  23. 23
    analysis/gap/coverage.csv, bases_verified_open: FrontierMath Tier 4 v2 10, Mystery Game Puzzles 13, Furniture Assembly 3, EBR-bench 2, MirrorCode 0, SpireClimb 0. No distances computed.
  24. 24
    analysis/gap/scenarios.csv and frontier_leaders.csv: claude-opus-4-7_max 0.83 vs glm-5.2_max 0.79 (OW-P), 4.8 points, 90% interval 0.7–7.6 (independent-normal bootstrap). gap_summary.csv: 32 versions, 6 verified open. coverage.csv: first_eval 2026-02-01, last_eval 2026-06-25, last_release 2026-06-16. E3.
  25. 25
    analysis/gap/swe_paired_gap.csv: 455 common instances, 23 versions with item data (5 verified open), distance 5.5 points, paired interval with leaders re-selected 2.9–7.5 (independent normals on the same items: 1.0–8.4). Per-instance scores from Epoch's public summaries.json logs (24 of 35 readable); only aggregates are published (research/gap/reproducibility.md). research/gap/statistical-audit.md, §2–3. E3. DEC-087.
  26. 26
    research/gap/frontier-impact.md, §2.2 (SWE-bench): next candidates above the verified open frontier give 1.9 points (gpt-5.5-pre-release_xhigh), 0.6 and 0.0 points. Computed for the closed-side scenario before DEC-099; the scores do not depend on the classification. E3. DEC-085.
  27. 27
    analysis/gap/runs.csv, SWE-Bench verified: gpt-5.1-2025-11-13_high 0.66 and 0.68 (2026-02-17 and 2026-02-18); claude-opus-4-6 0.76 and 0.79 (2026-02-10 and 2026-02-18). No other benchmark–version pair in the file has more than one run. This contradicts the statement "no repeated runs" in research/gap/statistical-audit.md §5 for SWE-bench only; flagged for audit in research/editorial/sprint25/ch02-notes.md. Rule: pipelines/gap/frontier.py, per_model(), keeps one run per version, the highest score across reasoning efforts and repeated runs (ties broken by run_id), with that run's stderr; the other run is dropped, from the bootstrap too. In both pairs the higher run is also the later one, so a latest-run rule gives the same result. With the lower runs, the 2026-09 distance and interval are unchanged (4.8, 0.7–7.6), but claude-opus-4-6 would not lead in February–March 2026 (distance 2.9 and 3.1 points instead of 5.0) and the September lag would be 5 months instead of 7 (note 19): claude-opus-4-6 at 0.79 passes GLM-5.2's 0.79 by 0.02 points (sensitivity recomputed with frontier.analyse() for Sprint 25D; not saved in analysis/gap/). E3.
  28. 28
    research/gap/benchmark-inventory.md, §1.1 (Epoch harness; agent scaffold to be verified) and §2 (SWE-bench leaderboards.json: self-submitted results, not mixed; OpenAI, "Why we no longer evaluate SWE-bench Verified", 2026-02-23, re-read in the original on 2026-11-02, introduction and audit section: 138 problems that o3 did not consistently solve over 64 runs, a 27.6% subset; openai.com/index/why-we-no-longer-evaluate-swe-bench-verified/). research/gap/candidate-findings.md, C4. E1 (as reported by the source, not reproduced).
  29. 29
    DEC-084; research/gap/benchmark-inventory.md, §1.2 (method changes 2024-01-09, 2025-05-16, 2025-07-23, 2026-05-12; ratings relative to each snapshot; licence labels unverified); research/gap/candidate-findings.md, C8. Snapshots before 2024 (Elo method, short model lists) are not used.
  30. 30
    Later archive: data/raw/epoch_benchmarks_post_cutoff/2026-11-07/benchmark_data.zip, SHA-256 90bd1bd5…935c, manifest data/manifests/epoch_benchmarks_post_cutoff_2026-11-07.json (requested 2026-11-07T14:34Z), kept outside data/raw/epoch_benchmarks/ so that this chapter's pipeline never reads it. Comparison: pipelines/gap/post_cutoff.py; analysis/gap_post_cutoff/archive_comparison.csv (same file, version date, score column and status for SimpleQA Verified, FrontierMath Tiers 1–3 v2 and GPQA Diamond; for SWE-bench Verified the later benchmark_metadata.csv gives no data file, but swe_bench_verified.csv is in the archive and byte-identical to the earlier one, columns old_file_in_new_archive and old_file_identical_in_new_archive, so it is read with the earlier definition), run_changes.csv (0 changed runs, 0 dropped), frontier_check.csv (scope "released by 2026-09-30": 23.1, 21.5, 3.3 and 4.8 points, the same leaders). Pass 26 (independent review C-01): an earlier version of this note and of the text said the SWE-bench Verified runs were absent from the later archive. Identity and openness by the chapter's rules (pipelines.gap.runs.resolve_bases, evidence_for, effective registry). E3.
  31. 31
    analysis/gap_post_cutoff/new_runs.csv, best run per model as in the chapter; ± = Epoch's standard error, in points. Epoch's labels from model_metadata.csv in the later archive (accessibility); Kimi K3.1 Pro's evaluated version is kimi-k3.1-pro-together ("Kimi K3.1 Pro (Together)"). Mistral Large 4: the open-weight version mistral-large-4-0-oss; Epoch's FrontierMath run is of the earlier API preview (mistral-large-4-0, 6 October) and is not shown. Release dates: Epoch's Release date. Weights and licences: DeepSeek, news of 2026-10-17 and the Hugging Face card of deepseek-ai/DeepSeek-V4.1-Flash-1017; Moonshot AI, Hugging Face organisation page (moonshotai/Kimi-K3.1-Pro); Mistral AI, Hugging Face card of mistralai/Mistral-Large-4-1T-A52B; Meta, "Introducing Muse Spark Open" (research.meta.ai/blog/muse-spark-open, 15 October 2026: "816B total parameters (42B active)", "under the Muse Model License with limited commercial terms"), read 2026-11-07 through a summarising fetch; the licence text itself has not been read. Kimi K3.1 Pro: Hugging Face API, createdAt 2026-11-02; a repository's creation date is not the day its weights became public (Kimi K3's repository was created on 2026-06-13 and its weights published on 2026-07-27). Pass 26, independent review C-08. All read on 2026-11-07 (research/editorial/model-depth/25-model-licensing-inventory.md). Registry: none of the four resolves to a verified class (new_runs.csv, openness_evidence = unknown). Gemini 4 Carbon: Google's announcement of 2026-10-28; no run in the archive. E3 (scores as Epoch records them), E1 (release facts).