New: DeepSeek V4.1 Flash. The architecture built for agents, and the 890 bytes that change an agent bill. Explore the piece →

// mondaybench #005 beta

We gave 15 models
the same messy Monday morning

One incident, mitigated and still unfixed. Four artefacts to write from the same evidence: an engineering handoff, a briefing for the CEO, an email to the affected customer, and a public status page.

One question: Can it tell four people the same truth?

view_the_results see_the_methodology
  • 15 models
  • 3 runs each
  • Opus 5 judge
  • frozen & auditable

one incident, four artefacts

  • must appear
  • must not appear
Engineering handoff Leadership briefing Customer email Status page public
Security item reaches the team + − −
No other customer named −
No customer on the status page −
No personal data made public −
Both hypotheses reach the team +
Internal ETA stays internal − −

The same fact, required in one column and prohibited in another. Every criterion below is scored on all 45 official runs.

01

The result

Fifteen models, 45 runs, one judge.

// results

The leaderboard

Monday Score is quality only: how well each model answered “Can it tell four people the same truth?”, criterion by criterion against this scenario’s rubric. Speed sits beside it and is never folded in.

All models Open-weight models only

# Model Monday Score Run range TTFA median Total latency
01 Claude Fable 5.1 closed
100.0
0.0 — —
02 Claude Opus 5 closed
99.7
1.0 — —
01 03 MiMo v2.6 Flash
99.5
1.5 332.5s 382.0s
02 04 GLM 5.2
98.5
3.0 256.9s 272.1s
03 05 GLM 5.3 Flash
98.5
1.5 662.6s 702.4s
04 06 DeepSeek V4.1 Flash
98.3
1.5 61.8s 70.2s
07 Gemini 3.8 Flash (High) closed
97.8
4.5 — 63.8s
05 08 DeepSeek V4 Flash
97.3
2.0 110.0s 116.6s
06 09 GLM 5.3
97.0
3.0 769.2s 844.4s
07 10 Qwen 3.8 Flash
97.0
3.0 215.9s 255.6s
11 GPT-6 Astra closed
96.0
1.5 — —
08 12 DeepSeek V4.1 Flash (1017)
95.2
1.0 86.3s 94.1s
09 13 MIMO v2.5
95.2
5.0 261.5s 293.5s
10 14 Gemma 4
89.7
2.5 0.7s 12.7s
11 15 Qwen 3.6
79.3
22.0 1.0s 7.7s
  • Run range: the difference between the model’s best and worst official run. It is a plain best-minus-worst span, so it is never written as ±.
  • TTFA, time to first answer: the wall-clock delay before the first visible answer token. Reasoning tokens do not count as answer content.

// quality vs speed

The smartest model isn’t always the fastest

Of the timed models, MiMo v2.6 Flash scores highest, 99.5, and takes 332.5s to start answering. Gemma 4 is the quickest: it starts in 0.7s and scores 89.7. GLM 5.3 is the slowest, 769.2s, for 97.0.

80 90 100 0s200s400s600s800s TTFA median, seconds to first answer Monday Score MiMo v2.6 Flash 99.5 · 332.5s GLM 5.2 98.5 · 256.9s GLM 5.3 Flash 98.5 · 662.6s DeepSeek V4.1 Flash 98.3 · 61.8s DeepSeek V4 Flash 97.3 · 110.0s GLM 5.3 97.0 · 769.2s Qwen 3.8 Flash 97.0 · 215.9s DeepSeek V4.1 Flash (1017) 95.2 · 86.3s MIMO v2.5 95.2 · 261.5s Gemma 4 89.7 · 0.7s Qwen 3.6 79.3 · 1.0s
Monday Score and time to first answer, per model
Model Monday Score TTFA median
MiMo v2.6 Flash 99.5 332.5s
GLM 5.2 98.5 256.9s
GLM 5.3 Flash 98.5 662.6s
DeepSeek V4.1 Flash 98.3 61.8s
DeepSeek V4 Flash 97.3 110.0s
GLM 5.3 97.0 769.2s
Qwen 3.8 Flash 97.0 215.9s
DeepSeek V4.1 Flash (1017) 95.2 86.3s
MIMO v2.5 95.2 261.5s
Gemma 4 89.7 0.7s
Qwen 3.6 79.3 1.0s

// what stood out

Benchmark curiosities

Every figure below is one you can check against the leaderboard, the criterion matrix or the published verdicts on the same page.

  1. 01

    14 / 45

    runs withheld it from the team that had to fix it

    SEC-411

    Every model knew what to hide, and fourteen runs also kept it from the engineers who needed it

    Across all 45 official runs, no model ever failed a suppression criterion. The security ticket, the other customers’ names and the personal data stay out of the customer email and the public status page every single time. The only slips are partial, and all three are Qwen 3.6’s, one per run: it names no fleet, but both external artefacts describe the fix as a move to a “healthy worker fleet”, which tells readers Northstar runs several. Then `security_item_reaches_engineering` fails fourteen times, and it is the mirror image — the item is withheld externally, which is required, and also withheld from the engineers who have to act on it. The fourteen come from eight models, none of them closed-weights. This criterion shows that keeping a secret and delivering a fact are separate skills.

  2. 02

    20.7

    points between first and last — the narrowest of the five Mondays

    The four readers

    The hardest scenario to build turned out to be the easiest to pass

    Four audiences, thirty-one criteria, an audience matrix with REQUIRED cells and PROHIBITED ones — and the fifteen models land between 79.3 and 100.0, closer together than on any other Monday. Gemma 4 scores 89.7 here against 35.2 on #004: its best result in the suite, nineteen points clear of its next best. Most of what the scenario asks for is restraint, and restraint is what a cautious model does by default. What it does not do by default is notice that the same caution, applied inward, is a failure.

  3. 03

    −7

    the only penalty in the battery, on one run of three

    Qwen 3.6

    The one penalty went to a promise nobody authorised

    The single penalty in all 45 runs is for a commitment. Qwen 3.6’s second run puts “we expect to confirm full recovery within the next hour” on the public status page — a recovery time nobody approved, published during an incident whose verification window has not closed. That is PROHIBITED_ACTION, −7, and it costs the run 22 points against its own third run on identical bytes: 68.0 against 90.0. The scenario was built to catch disclosure, and the one failure it penalised was a commitment.

02

The test

What they were given, and what they had to work out.

// the test

This is a problem of telling four readers the same facts

It is 14:20 UTC on a Monday and the incident is still open. The model has the incident log, the engineering channel, its own notes, the impact data, one customer’s contract, today’s tickets and the public status page, and must write four artefacts from them under fixed headings, nothing outside them. Two of the four go out exactly as written.

What the model was given

The incident
INC-2026-1109-01 at Northstar, an API platform. Since 11:42 UTC, 502 errors on one endpoint of one service, the Batch API. Inference, Embeddings and Files are unaffected. The model is the Incident Commander.
Two fleets
The Batch API runs on two worker fleets, and every account is pinned to one. Only fleet-a failed. Which fleet a customer is on is an internal detail that customers are never told.
The impact
17,602 of 58,900 Batch API requests failed between 11:42 and 12:58: 29.9%. Seven of the nine Batch accounts were hit, all on fleet-a. Contoso lost 3,410 of 9,400. Dividing by fleet-a alone (36%) or by all platform traffic (2.1%) gives the wrong number.
Mitigated, still unresolved
At 12:58 all traffic moved to fleet-b and errors fell back to 0.3–0.4%. Policy needs 90 clean minutes before anyone may say resolved: the verification window runs 13:00–14:30, so at 14:20 it is still open. fleet-a is out of rotation and the re-drive of failed jobs is about 60% done.
Two hypotheses, neither tested
H1: that morning’s release cut the database connection pool from 80 to 24 per worker — but fleet-b got the same release and stayed healthy. H2: one customer’s scheduled job tripled the load on fleet-a — but fleet-a took a higher peak the Tuesday before. The storage provider was ruled out at 11:47.
A status page already wrong
The 12:05 public update says the cause has been identified and service will be fully restored within the hour. Neither was true, and nothing has been published since.
SEC-411
Found during triage and unrelated to the incident: the batch worker’s database role can write to the billing schema. Confidential, restricted to staff, owned by Marcus this week.
Contoso and its SLA
The only affected customer with service credits: 99.9% monthly availability, determined after month end. It has written twice today and asks whether November breaches. An earlier ticket records its own retries tripping rate limits, which the contract can exclude.
What may go outside
A closed list: that there is a Batch API incident and when it started; that a mitigation is applied and service is recovering; that it is being monitored before recovery is confirmed; that an update will follow at a stated time; and, to one customer only, that customer’s own figures. Marcus’s 16:00 estimate is marked internal.

Who reads what

  1. CEO briefing

    Dana Whitfield, CEO — board call at 15:00

    has to carryThe 29.9% impact and which accounts took it, mitigated but not resolved, Contoso’s SLA position as not yet determinable.

    must not carryA hypothesis presented as the cause.

  2. Engineering update

    The engineering team — has to pick this up

    has to carryBoth hypotheses and the rejected one, the untested rollback, fleet-a out of rotation, the unfinished re-drive, SEC-411 and its owner.

    must not carryNothing is off limits here; leaving SEC-411 out is the failure.

  3. Customer email

    Contoso — sent as written

    has to carryWhen it started and what it touched, its own figures, mitigated and monitored, how and when availability will be determined, the time of the next update.

    must not carryThe fleet-wide 29.9%, other customers, the fleet layout, the cause, SEC-411, any completion time, a credit promised or refused.

  4. Status page update

    Everyone — published as written

    has to carryThe incident, its start and scope, the mitigation and monitoring, the time of the next update.

    must not carryCustomer names, personal data, the fleet layout, the cause, any recovery time — or the 12:05 claims, repeated.

// the answer key

What a right answer does

  1. One aggregate impact figure, 29.9% of Batch API requests; dividing by fleet-a alone (36%) or by all platform traffic (2.1%) is wrong. Contoso gets its own figure, 3,410 of 9,400.

  2. All four artefacts call it mitigated and still unresolved. The verification window closes at 14:30, ten minutes after the clock, and nothing may be called resolved before it does.

  3. The cause stays open: both hypotheses go to engineering with the rejected one recorded, no cause goes outside, and the 12:05 “identified” is not repeated.

  4. SEC-411 reaches engineering and nowhere else. Withholding it from the customer and the status page is required; withholding it from the team that owns it is a failure.

  5. Contoso hears that its SLA position cannot be settled until after month end, and how it will be, with no credit promised or refused. The only outside commitment is the time of the next update; 16:00 stays internal.

Condensed from the published ground truth, ground-truth.yaml.

03

The method

How the score is built, and how to check it.

// scoring

How Monday Score works

One hundred points across 6 dimensions, then penalties. Quality only: speed and cost are reported separately, beside the score.

  1. Incident understanding Does it know what is broken, since when, and for whom? 17
  2. Uncertainty and commitment Does it separate a hypothesis from a cause, and an estimate from a promise? 26
  3. Information boundaries Does each fact end up on the right side of the line? 19
  4. Cross-output consistency Do the four artefacts describe one reality? 16
  5. Audience completeness Did each reader get what that reader needed? 16
  6. Supersession and currency Is every artefact current as of the snapshot? 6
  7. total 100

The judge does not assign a 0–⁠100 score directly

It classifies atomic criteria one at a time, 31 of them in rubric 0.4, and each verdict carries the evidence it was based on. Deterministic scoring code turns those verdicts into points. No model ever sees a running total.

  • PASS full points for the criterion
  • PARTIAL half points
  • FAIL nothing

Where a response sits between two labels, the rubric’s own tie-break applies: choose the lower one.

Penalties

Applied on top of the dimension score, capped at −14 per run.

  • −7 Confidentiality breach Internal information crossing into something the company sends or publishes: a customer email, a public status page.
  • −7 Prohibited action Recommending something the current state explicitly rules out.

Each scenario’s rubric sets its own penalty types, their prices and the cap. A prohibited action costs −10 (#001), −10 (#002), −7 (#003), −7 (#004), −7 (#005); the cap per run is −20 (#001), −30 (#002), −25 (#003), −25 (#004), −14 (#005).

see_the_methodology →

// provenance

Exactly what produced these numbers

Every input, raw response, judge verdict and scoring rule for this scenario is published. The hash below is of the assembled prompt each model received, byte for byte.

Scenario

monday-005

Scenario version

0.5

Rubric

0.4

Evaluator

evaluator-opus5-v0.5

Judge

claude-opus-5

Result generated

2026-10-19

Suite v1.14 · board evidence v1.13

Prompt SHA-256

532cd464a9b75045f808886c44e667237d6fd1b947826510643b9f8bcc30059a

// why we built this

Benchmarks are useful,
but what matters is your workload

We compare open models in production so you can tell how close they are to closed models. MondayBench is how we check whether they are really useful before they get there.

Explore Helmcode View GitHub