48 Slack messages · 7 emails · 2 meetingsthree of them change what is already true
metrics.csv Umbrella’s anomaly — in the metrics, mentioned by nobody
01
The result
Fifteen models, 45 runs, one judge.
// results
The leaderboard
Monday Score is quality only: how well each model answered “Can it figure out what actually matters?”, criterion by criterion against this scenario’s rubric. Speed sits beside it and is never folded in.
All modelsOpen-weight models only
monday-001 · 15 models, 3 runs each
#
Model
Monday Score
Run range
TTFA median
Total latency
Show detail for
01
GPT-6 Astraclosed
100.0
0.0
—
—
Individual runs
Run 1100.0
Run 2100.0
Run 3100.0
Speed
TTFA
—
Total latency
—
Median output tokens
—
Dimension breakdown
Action detection 25/25
State reconstruction 25/25
Prioritization 15/15
Data reasoning 15/15
Uncertainty 10/10
Noise resistance 10/10
What it missed
Full marks on every criterion.
What it got wrong
No penalties and no false actions in any run.
02
Claude Opus 5closed
98.5
0.0
—
—
Individual runs
Run 198.5
Run 298.5
Run 398.5
Speed
TTFA
—
Total latency
—
Median output tokens
2,952
Dimension breakdown
Action detection 25/25
State reconstruction 25/25
Prioritization 15/15
Data reasoning 13.5/15
Uncertainty 10/10
Noise resistance 10/10
What it missed
Judge’s wording below, unedited.
Errors read against traffic−1.5 pts forgone
umbrella_error_rate
R1 PARTIALR2 PARTIALR3 PARTIAL
Compares Umbrella's 163 errors with other accounts / says it is high relative to volume, but does not give Umbrella's 94,217 request denominator, an error rate, or the Stark high-traffic comparison; lower-label rule => PARTIAL.
What it got wrong
No penalties and no false actions in any run.
0103
GLM 5.3 Flash
98.0
1.5
118.8s
132.4s
Individual runs
Run 198.5
Run 297.0
Run 398.5
Speed
TTFA
118.8s
Total latency
132.4s
Median output tokens
5,603
Dimension breakdown
Action detection 25/25
State reconstruction 25/25
Prioritization 15/15
Data reasoning 13.5/15
Uncertainty 9.5/10
Noise resistance 10/10
What it missed
Judge’s wording below, unedited.
Errors read against traffic−1.5 pts forgone
umbrella_error_rate
R1 PARTIALR2 PARTIALR3 PARTIAL
'1821ms (vs ~300 for everyone else)' quotes the peers' latency and 'Stark/Contoso error rates negligible' reaches for the rate idea, but no request volume and no rate figure - the 163-against-94,217 reading is never made
Friday conversion unverifiable−0.5 pts forgone
friday_conversion_unknown
R1 PASSR2 PARTIALR3 PASS
'Julia saw weak conversion Friday but suspects noisy traffic; no owner assigned' - flagged, but no statement that the provided data cannot settle it
What it got wrong
No penalties and no false actions in any run.
04
Claude Fable 5.1closed
97.5
1.5
—
—
Individual runs
Run 197.0
Run 298.5
Run 397.0
Speed
TTFA
—
Total latency
—
Median output tokens
2,352
Dimension breakdown
Action detection 25/25
State reconstruction 25/25
Prioritization 15/15
Data reasoning 13.5/15
Uncertainty 9/10
Noise resistance 10/10
What it missed
Judge’s wording below, unedited.
Errors read against traffic−1.5 pts forgone
umbrella_error_rate
R1 PARTIALR2 PARTIALR3 PARTIAL
compares Umbrella's raw error count and latency with other customers but never cites the 94,217 request volume, an error rate, or Stark's higher raw count on far higher traffic
compares with other customers without connecting the count to traffic volume
Friday conversion unverifiable−1 pts forgone
friday_conversion_unknown
R1 PARTIALR2 PASSR3 PARTIAL
nobody verified the Friday conversion drop; filed under needs-more-information
POST-AUDIT CORRECTION PASS -> PARTIAL (experiments/fable-5.1-full-mondaybench/AUDIT.md section 5). Original reason: flagged as unverifiable from the data
What it got wrong
No penalties and no false actions in any run.
0205
GLM 5.3
97.5
6.0
181.6s
202.8s
Individual runs
Run 1100.0
Run 294.0
Run 398.5
Speed
TTFA
181.6s
Total latency
202.8s
Median output tokens
6,861
Dimension breakdown
Action detection 23.8/25
State reconstruction 25/25
Prioritization 14.2/15
Data reasoning 14.5/15
Uncertainty 10/10
Noise resistance 10/10
What it missed
Judge’s wording below, unedited.
Umbrella action−1.2 pts forgone
umbrella_action
R1 PASSR2 PARTIALR3 PASS
detected and actioned for today ('Raise with engineering/CS today'), no root cause claimed - but ranked #4, below a holding reply to Initech and below the churn analysis, so it is under-prioritised; between PASS and PARTIAL the rubric's tie-break takes the lower
Umbrella and churn prioritised−0.8 pts forgone
umbrella_and_churn_high
R1 PASSR2 PARTIALR3 PASS
churn is appropriately placed at #3; Umbrella is at #4 behind a routine email reply, so only one of the two is properly prioritised
Errors read against traffic−0.5 pts forgone
umbrella_error_rate
R1 PASSR2 PASSR3 PARTIAL
compares Umbrella with the other accounts on latency ('every other account: 267-341 ms') and notes they look healthy, but never reads the 163 errors against Umbrella's 94,217 requests, gives no error rate, and does not note that Stark carries more raw errors on far higher traffic
What it got wrong
No penalties and no false actions in any run.
06
Gemini 3.8 Flash (High)closed
95.3
2.5
—
13.2s (route-timed)
Individual runs
Run 197.0
Run 294.5
Run 394.5
Speed
TTFA
—
Total latency
13.2s · route-timed
Median output tokens
—
Dimension breakdown
Action detection 25/25
State reconstruction 25/25
Prioritization 13.3/15
Data reasoning 12/15
Uncertainty 10/10
Noise resistance 10/10
What it missed
Judge’s wording below, unedited.
Errors read against traffic−3 pts forgone
umbrella_error_rate
R1 FAILR2 FAILR3 FAIL
'elevated errors (163)' with no reference to 94,217 requests, an error rate, or Stark Industries' higher raw errors on higher traffic.
Raw error count is treated as alarming without reference to traffic.
Umbrella and churn prioritised−1.7 pts forgone
umbrella_and_churn_high
R1 PASSR2 PARTIALR3 PARTIAL
Umbrella is #2 HIGH, but churn is #4 'MEDIUM... (Due Tuesday)', ranked below the #3 Initech pricing reply.
Only Umbrella is appropriately prioritised; churn ahead of the board meeting sits below a holding reply.
What it got wrong
No penalties and no false actions in any run.
0307
DeepSeek V4.1 Flash
95.0
6.0
39.3s
44.1s
Individual runs
Run 191.0
Run 297.0
Run 397.0
Speed
TTFA
39.3s
Total latency
44.1s
Median output tokens
8,154
Dimension breakdown
Action detection 23.8/25
State reconstruction 25/25
Prioritization 14.2/15
Data reasoning 12/15
Uncertainty 10/10
Noise resistance 10/10
What it missed
Judge’s wording below, unedited.
Errors read against traffic−3 pts forgone
umbrella_error_rate
R1 FAILR2 FAILR3 FAIL
The response reports Umbrella's anomaly but does not contextualize the 163 errors against request volume or another account.
No usable error-rate comparison is made.
Umbrella action−1.2 pts forgone
umbrella_action
R1 PARTIALR2 PASSR3 PASS
Umbrella is noticed, but the investigation is not promoted strongly enough into today's action list.
The response recognizes the Umbrella anomaly but gives it weaker operational treatment than the PASS band requires.
Umbrella and churn prioritised−0.8 pts forgone
umbrella_and_churn_high
R1 PARTIALR2 PASSR3 PASS
Churn is actionable, while Umbrella is not kept at the same high-priority level.
Only one of the two required high-priority investigations is consistently elevated.
What it got wrong
No penalties and no false actions in any run.
0408
Qwen 3.8 Flash
92.5
6.0
72.2s
92.0s
Individual runs
Run 193.5
Run 295.0
Run 389.0
Speed
TTFA
72.2s
Total latency
92.0s
Median output tokens
9,253
Dimension breakdown
Action detection 25/25
State reconstruction 25/25
Prioritization 11.5/15
Data reasoning 12.5/15
Uncertainty 8.5/10
Noise resistance 10/10
What it missed
Judge’s wording below, unedited.
No false priorities−2.7 pts forgone
no_false_priorities
R1 PARTIALR2 PARTIALR3 FAIL
two cleanup items outrank the churn analysis: the resolved Acme incident follow-up at #3 and the Analytics v2 task cleanup at #4, which closes the launch-email task Julia already owns
Errors read against traffic−2.5 pts forgone
umbrella_error_rate
R1 FAILR2 PARTIALR3 FAIL
the three figures are given in isolation: no request volume, no rate, no reference to any other customer
Friday conversion unverifiable−1.5 pts forgone
friday_conversion_unknown
R1 PARTIALR2 PARTIALR3 PARTIAL
'The team suspected noisy traffic, but it has not been investigated' - flagged, but no statement that the provided data cannot settle it
Umbrella and churn prioritised−0.8 pts forgone
umbrella_and_churn_high
R1 PASSR2 PASSR3 PARTIAL
Umbrella is well placed at #2, but the churn analysis is #6 of eight, below two task-cleanup items and the Initech reply
What it got wrong
No penalties and no false actions in any run.
0509
DeepSeek V4 Flash
90.8
8.0
55.4s
63.3s
Individual runs
Run 195.5
Run 287.5
Run 389.5
Speed
TTFA
55.4s
Total latency
63.3s
Median output tokens
4,871
Dimension breakdown
Action detection 22.7/25
State reconstruction 25/25
Prioritization 12.7/15
Data reasoning 12/15
Uncertainty 8.5/10
Noise resistance 10/10
What it missed
Judge’s wording below, unedited.
Errors read against traffic−3 pts forgone
umbrella_error_rate
R1 FAILR2 FAILR3 FAIL
the three figures are given in isolation: no request volume, no rate, no reference to any other customer
Umbrella action−2.3 pts forgone
umbrella_action
R1 PASSR2 PARTIALR3 PARTIAL
only under 'unclear': 'Umbrella metrics look anomalous... Needs an explanation' - never enters today's list
Umbrella and churn prioritised−1.7 pts forgone
umbrella_and_churn_high
R1 PASSR2 PARTIALR3 PARTIAL
churn actionable at #4; Umbrella appears only under 'unclear'
Friday conversion unverifiable−1.5 pts forgone
friday_conversion_unknown
R1 PARTIALR2 PARTIALR3 PARTIAL
'possibly noise, not yet investigated'; 'real or noise? Needs a quick check' - flagged, but no statement that the provided data cannot settle it
No false priorities−0.7 pts forgone
no_false_priorities
R1 PASSR2 PARTIALR3 PASS
the #3 task-cleanup bundle, which closes the resolved Acme task, is ranked above the churn analysis
What it got wrong
No penalties and no false actions in any run.
0610
DeepSeek V4.1 Flash (1017)
90.3
11.0
35.4s
40.5s
Individual runs
Run 184.5
Run 295.5
Run 391.0
Speed
TTFA
35.4s
Total latency
40.5s
Median output tokens
6,821
Dimension breakdown
Action detection 21.3/25
State reconstruction 24.2/25
Prioritization 13.3/15
Data reasoning 12/15
Uncertainty 9.5/10
Noise resistance 10/10
What it missed
Judge’s wording below, unedited.
Errors read against traffic−3 pts forgone
umbrella_error_rate
R1 FAILR2 FAILR3 FAIL
Cites '163 API errors' with no request volume, error rate, or Stark comparison.
Raw error count with no reference to traffic.
Umbrella action−2.3 pts forgone
umbrella_action
R1 PARTIALR2 PASSR3 PARTIAL
Umbrella appears only as a sub-clause of the churn item: 'Include/assign a check on the Umbrella usage/error/latency anomaly'; otherwise listed as a 'Metrics standout' and in the unclear section.
Notices the anomaly and asks for a check, but folds it into churn analysis rather than treating it as its own investigation today, i.e. under-prioritized.
Umbrella and churn prioritised−1.7 pts forgone
umbrella_and_churn_high
R1 PARTIALR2 PASSR3 PARTIAL
Churn is standalone #2; Umbrella is only 'Include/assign a check on the Umbrella... anomaly' inside the churn item.
Churn is appropriately prioritized but Umbrella is buried inside another item, so only one is prominent.
Contoso action−1.3 pts forgone
contoso_action
R1 PARTIALR2 PASSR3 PASS
#1 'Contoso renewal — hard deadline 11:00. Work with Noah/Marcus to get written confirmation... Do not guess.' No statement anywhere that the signature/renewal slips if unresolved.
Top priority with the 11:00 deadline, but misses the renewal-blocking consequence, which the PARTIAL definition names.
Renewal blocked−0.8 pts forgone
contoso_blocked
R1 PARTIALR2 PASSR3 PASS
'Contoso legal needs written confirmation by 11:00 Monday that admin-deleted data is removed from DR backups... Noah flagged this as the only remaining concern'.
Identifies the outstanding legal question but never states that the signature moves to the next approval window if unresolved.
Friday conversion unverifiable−0.5 pts forgone
friday_conversion_unknown
R1 PASSR2 PARTIALR3 PASS
Only in no-action section: 'Conversion dip was noticed Friday and may be noisy Friday traffic; worth checking next week.' Not listed under unclear.
Does not assert the drop as fact, but never says it cannot be verified from the provided data.
What it got wrong
No penalties and no false actions in any run.
0711
MIMO v2.5
88.2
6.5
42.4s
64.1s
Individual runs
Run 191.0
Run 289.0
Run 384.5
Speed
TTFA
42.4s
Total latency
64.1s
Median output tokens
2,150
Dimension breakdown
Action detection 20.5/25
State reconstruction 25/25
Prioritization 11.2/15
Data reasoning 13/15
Uncertainty 8.5/10
Noise resistance 10/10
What it missed
Judge’s wording below, unedited.
Umbrella action−3.5 pts forgone
umbrella_action
R1 PARTIALR2 PARTIALR3 PARTIAL
only under 'Unclear': 'Worth investigating - is there an active issue, or was this just a quiet week?' - never enters today's list
Umbrella and churn prioritised−2.5 pts forgone
umbrella_and_churn_high
R1 PARTIALR2 PARTIALR3 PARTIAL
churn actionable in today's list; Umbrella appears only under 'Unclear'
Errors read against traffic−2 pts forgone
umbrella_error_rate
R1 PARTIALR2 PARTIALR3 FAIL
the three figures are given in isolation: no request volume, no rate, and no reference to any other customer
Friday conversion unverifiable−1.5 pts forgone
friday_conversion_unknown
R1 PARTIALR2 PARTIALR3 PARTIAL
'flagged as potentially noisy Friday traffic. Worth revisiting next week' - not repeated as fact, but no data-gap statement
No false priorities−1.3 pts forgone
no_false_priorities
R1 PASSR2 PARTIALR3 PARTIAL
Globex, which the source says has no urgency, is #4, above the Umbrella investigation
Churn action−1 pts forgone
churn_action
R1 PASSR2 PASSR3 PARTIAL
the board deadline is stated ('Need a fuller breakdown for Tuesday's board meeting'), but the analysis never enters the action table - it is filed under 'no action required from you' as 'Analysis task is yours for Tuesday'
What it got wrong
No penalties and no false actions in any run.
0812
GLM 5.2
84.3
16.0
2.7s
6.9s
Individual runs
Run 187.0
Run 275.0
Run 391.0
Speed
TTFA
2.7s
Total latency
6.9s
Median output tokens
1,241
Dimension breakdown
Action detection 20.3/25
State reconstruction 25/25
Prioritization 11.2/15
Data reasoning 10/15
Uncertainty 7.8/10
Noise resistance 10/10
What it missed
Judge’s wording below, unedited.
Umbrella action−4.7 pts forgone
umbrella_action
R1 PARTIALR2 FAILR3 PARTIAL
Umbrella is never mentioned; the CSV is opened only to check that Initech is healthy
Umbrella and churn prioritised−2.5 pts forgone
umbrella_and_churn_high
R1 PARTIALR2 PARTIALR3 PARTIAL
churn well placed at #3, but the Umbrella investigation is buried at #6
Errors read against traffic−2 pts forgone
umbrella_error_rate
R1 PARTIALR2 FAILR3 PARTIAL
no Umbrella reasoning at all
Umbrella anomaly detected−1.7 pts forgone
umbrella_detected
R1 PASSR2 FAILR3 PASS
the anomaly is missed entirely - the only customer row it reads is Initech's, and it reads it correctly as healthy
Friday conversion unverifiable−1.5 pts forgone
friday_conversion_unknown
R1 PARTIALR2 PARTIALR3 PARTIAL
'Is it noise or a real signal? Needs a look at the dashboard' - uncertainty flagged, but no statement that the provided data cannot settle it
No false priorities−1.3 pts forgone
no_false_priorities
R1 FAILR2 PASSR3 PASS
two false priorities outrank the Umbrella investigation: the resolved Acme task close-out at #4 and Globex at #5, which the source says has no urgency
Umbrella metrics cited−1.3 pts forgone
umbrella_metrics
R1 PASSR2 FAILR3 PASS
no Umbrella figures
Umbrella cause left open−0.7 pts forgone
umbrella_unknown_cause
R1 PASSR2 PARTIALR3 PASS
invents no cause, but expresses no uncertainty either - Umbrella is absent
What it got wrong
No penalties and no false actions in any run.
0913
Qwen 3.6
82.8
5.0
0.8s
6.2s
Individual runs
Run 182.5
Run 280.5
Run 385.5
Speed
TTFA
0.8s
Total latency
6.2s
Median output tokens
1,036
Dimension breakdown
Action detection 21.5/25
State reconstruction 25/25
Prioritization 11.2/15
Data reasoning 12/15
Uncertainty 6.5/10
Noise resistance 10/10
What it missed
Judge’s wording below, unedited.
Umbrella action−3.5 pts forgone
umbrella_action
R1 PARTIALR2 PARTIALR3 PARTIAL
under 'does not require action': 'a potential churn risk, though not explicitly flagged as urgent... Worth keeping in mind' - noticed, but not work for today
Errors read against traffic−3 pts forgone
umbrella_error_rate
R1 FAILR2 FAILR3 FAIL
the 163 errors are never mentioned; no volume, no rate, no other customer's figures
Umbrella and churn prioritised−2.5 pts forgone
umbrella_and_churn_high
R1 PARTIALR2 PARTIALR3 PARTIAL
churn actionable at #2 HIGH; Umbrella filed under 'does not require action'
Umbrella cause left open−2 pts forgone
umbrella_unknown_cause
R1 PARTIALR2 PARTIALR3 PARTIAL
no cause asserted as established, but no uncertainty expressed either, and Umbrella is absent from section 4
Friday conversion unverifiable−1.5 pts forgone
friday_conversion_unknown
R1 PARTIALR2 PARTIALR3 PARTIAL
the Friday conversion question is never raised: the claim is not repeated, but the gap is not flagged
No false priorities−1.3 pts forgone
no_false_priorities
R1 PASSR2 PARTIALR3 PARTIAL
Globex, which the source says has no urgency, is #2 High, above the churn analysis
What it got wrong
MATERIAL_HALLUCINATION-5 · Run 1
Strictly confidential until Wednesday's review / state that new tiers are under review and will be announced publicly soon
The source says pricing stays confidential until approved AND publicly announced; Wednesday is only the next review, and no announcement is scheduled at all. The error is written into the reply the user is told to send Initech, so it leaves the company.
MATERIAL_HALLUCINATION-5 · Run 2
Strictly confidential until Wednesday's review / state that pricing is under review and will be communicated officially soon
The source says pricing stays confidential until approved AND publicly announced; Wednesday is only the next review. A reader could conclude the tiers may be shared after Wednesday.
1014
MiMo v2.6 Flash
82.7
9.5
40.9s
81.6s
Individual runs
Run 180.5
Run 279.0
Run 388.5
Speed
TTFA
40.9s
Total latency
81.6s
Median output tokens
2,042
Dimension breakdown
Action detection 19.3/25
State reconstruction 23.3/25
Prioritization 11.3/15
Data reasoning 12/15
Uncertainty 8.3/10
Noise resistance 10/10
What it missed
Judge’s wording below, unedited.
Errors read against traffic−3 pts forgone
umbrella_error_rate
R1 FAILR2 FAILR3 FAIL
Cites 163 errors 'against every other customer' with no request volume, error rate or Stark comparison.
Raw count without reference to traffic.
Contoso action−2.7 pts forgone
contoso_action
R1 PASSR2 PARTIALR3 PARTIAL
'🔴 1. Contoso data retention confirmation — deadline 11:00 today ... €72k ARR at stake ... draft a response to Sarah before the deadline.'
Top priority with the 11:00 deadline, but never states that missing it pushes the renewal signature to the next approval window.
Umbrella action−2.3 pts forgone
umbrella_action
R1 PARTIALR2 PARTIALR3 PASS
Umbrella appears only under section 4 'unclear' ('Is this expected, being handled elsewhere, or a blind spot?'); absent from the today-priority list.
Notices the anomaly but does not make investigating it a today action, so it is under-prioritized.
No false priorities−2 pts forgone
no_false_priorities
R1 PARTIALR2 PARTIALR3 PARTIAL
'P1 — Close or update Acme task' ranked at the same P1 level as churn and above Umbrella (which is not in the list).
One resolved item (Acme task housekeeping) receives moderate priority over legitimate work.
Umbrella and churn prioritised−1.7 pts forgone
umbrella_and_churn_high
R1 PARTIALR2 PARTIALR3 PASS
Churn is P1 in today's list; Umbrella is only item 3 of the 'unclear' section.
Only churn is appropriately prioritized as an action for today.
Renewal blocked−1.7 pts forgone
contoso_blocked
R1 PASSR2 PARTIALR3 PARTIAL
'Commercial terms agreed. However, Sarah Mitchell sent a legal question'; only '€72k ARR at stake'.
Identifies the outstanding legal question but not the signature/approval-window consequence.
Friday conversion unverifiable−1 pts forgone
friday_conversion_unknown
R1 PARTIALR2 PARTIALR3 PASS
Section 1: 'Conversion metrics showed a drop on Friday ... Possibly noisy'; section 4: 'Julia noticed a significant conversion drop... It could be noisy traffic.'
Hedged and listed as unclear, but never says the drop cannot be verified from the provided data and partly states it as observed.
Umbrella cause left open−0.7 pts forgone
umbrella_unknown_cause
R1 PARTIALR2 PASSR3 PASS
'Is this expected, being handled elsewhere, or a blind spot?' — asks whether it is known/handled, never states the cause is unknown.
No invented cause, but the anomaly-versus-unknown-cause distinction is not explicit.
Analytics v2 action−0.7 pts forgone
analytics_action
R1 PASSR2 PARTIALR3 PASS
'🟠 2. Review Analytics v2 launch readiness (task due today)' ranked above churn and Umbrella, while stating 'Release moved to Wednesday'.
Understands Wednesday but gives the release disproportionate priority.
What it got wrong
MATERIAL_HALLUCINATION-5 · Run 1
'Do Pied Piper and Hooli (combined €27k churned ARR per Friday's finance snapshot) fully account for the 1.3pp move'
The finance snapshot gives €27k total churned ARR for the week; nothing attributes it to Pied Piper and Hooli. This invented attribution feeds directly into the churn breakdown the user must prepare for the board.
1115
Gemma 4
70.7
6.5
0.6s
5.5s
Individual runs
Run 168.5
Run 268.5
Run 375.0
Speed
TTFA
0.6s
Total latency
5.5s
Median output tokens
605
Dimension breakdown
Action detection 15.3/25
State reconstruction 23.3/25
Prioritization 12.5/15
Data reasoning 3/15
Uncertainty 6.5/10
Noise resistance 10/10
What it missed
Judge’s wording below, unedited.
Umbrella action−7 pts forgone
umbrella_action
R1 FAILR2 FAILR3 FAIL
Umbrella never mentioned; the metrics file is not used at all
Umbrella anomaly detected−5 pts forgone
umbrella_detected
R1 FAILR2 FAILR3 FAIL
anomaly missed
Umbrella metrics cited−4 pts forgone
umbrella_metrics
R1 FAILR2 FAILR3 FAIL
no supporting figures at all
Errors read against traffic−3 pts forgone
umbrella_error_rate
R1 FAILR2 FAILR3 FAIL
no metrics reasoning of any kind
Contoso action−2.7 pts forgone
contoso_action
R1 PARTIALR2 PARTIALR3 PASS
first item under High with the 11:00 deadline, but 'ready to renew pending a single legal clarification' states sequence, not jeopardy: nothing says what happens if 11:00 is missed
Umbrella and churn prioritised−2.5 pts forgone
umbrella_and_churn_high
R1 PARTIALR2 PARTIALR3 PARTIAL
churn is a High item; Umbrella is entirely absent
Umbrella cause left open−2 pts forgone
umbrella_unknown_cause
R1 PARTIALR2 PARTIALR3 PARTIAL
invents no cause, but expresses no uncertainty either - Umbrella is absent
Renewal blocked−1.7 pts forgone
contoso_blocked
R1 PARTIALR2 PARTIALR3 PASS
'pending a single legal clarification' - identifies the question, misses the signature or approval consequence
Friday conversion unverifiable−1.5 pts forgone
friday_conversion_unknown
R1 PARTIALR2 PARTIALR3 PARTIAL
'currently unconfirmed if this was a data anomaly or a trend' - uncertainty noted, data gap not flagged
What it got wrong
No penalties and no false actions in any run.
Run range: the difference between the model’s best and worst official run. It is a plain best-minus-worst span, so it is never written as ±.
TTFA, time to first answer: the wall-clock delay before the first visible answer token. Reasoning tokens do not count as answer content.
// quality vs speed
The smartest model isn’t always the fastest
Of the timed models, GLM 5.3 Flash scores highest, 98.0, and takes 118.8s to start answering. Gemma 4 is the quickest: it starts in 0.6s and scores 70.7. GLM 5.3 is the slowest, 181.6s, for 97.5.
Monday Score and time to first answer, per model
Model
Monday Score
TTFA median
GLM 5.3 Flash
98.0
118.8s
GLM 5.3
97.5
181.6s
DeepSeek V4.1 Flash
95.0
39.3s
Qwen 3.8 Flash
92.5
72.2s
DeepSeek V4 Flash
90.8
55.4s
DeepSeek V4.1 Flash (1017)
90.3
35.4s
MIMO v2.5
88.2
42.4s
GLM 5.2
84.3
2.7s
Qwen 3.6
82.8
0.8s
MiMo v2.6 Flash
82.7
40.9s
Gemma 4
70.7
0.6s
// what stood out
Benchmark curiosities
15 models, 45 runs, one judge. The patterns that stood out when reading the results.
01
GPT-6 Astra solved Monday, and the best open model came within two points
GPT-6 Astra scored 100.0 on all three runs, and Claude Opus 5 98.5 on all three. The best open model, GLM 5.3 Flash, was next at 98.0: three runs inside a point and a half of each other, with full marks on action detection, state reconstruction, prioritization and noise resistance.
Where it lost points It spotted the Umbrella anomaly but never fully reasoned about errors relative to request volume.
98.0/100
GLM 5.3 Flash
run 198.5
run 297.0
run 398.5
range1.5
Perfect on four of six dimensions
02
One run isn’t enough
A single run could make GLM 5.2 look like a 75-point model or a 91-point model. That is why every official MondayBench result uses three runs, and why the leaderboard reports the spread right next to the score.
DeepSeek V4 Flash
run 1: 95.5run 2: 87.5run 3: 89.5
95.5 / 87.5 / 89.5 range 8.0
GLM 5.2
run 1: 87.0run 2: 75.0run 3: 91.0
87.0 / 75.0 / 91.0 range 16.0
65100
03
Some models read the conversation but miss the business
Gemma understood much of the conversation, but missed the most important signal hidden in the customer metrics. Umbrella was never mentioned in any of its three runs: the metrics file was effectively ignored.
Data reasoning State reconstruction stayed high: it followed the thread, it just never opened the spreadsheet.
3/15
Data reasoning
Gemma 470.7
runs naming Umbrella0/3
04
Among the fastest, and the only one with hallucination penalties
Qwen 3.6 was one of the fastest models in the benchmark, but it was also the only model to receive material hallucination penalties.
The scenario states that proposed pricing stays confidential until it is approved and publicly announced. In two of three runs Qwen 3.6 rewrote that as confidentiality ending after Wednesday’s review, or as an announcement being imminent, and put it in the reply it told the user to send the customer. Its third run states the rule correctly.
2
penalties
ttfa0.8s
total6.2s
score82.8
02
The test
What they were given, and what they had to work out.
Rollback complete. Error rates are returning to normal.
#sales Fri 08:51 · Noah Williams
Contoso update: call went really well. Commercial terms are basically agreed.
#product Fri 09:36 · Sam Rivera
The export regression is more annoying than we thought. I’m proposing Wednesday instead of Monday.
#leadership Fri 11:42 · Daniel Foster
I’m slightly concerned about enterprise churn. The dashboard has it moving from 3.1% to 4.4%. … Would be good to understand what’s driving that before the board meeting next week.
tasks
Priority
Task
Owner
Due
Urgent
Investigate Acme API failures stale
You
Thursday
High
Confirm Contoso renewal next steps
You
Monday
High
Prepare Analytics v2 launch email stale
You
Monday
Medium
Analyse enterprise churn
You
Tuesday
inbox
Fri 16:18 · sarah@contoso.example
One remaining point before renewal
…our legal team flagged one remaining question regarding data retention. They need
written confirmation that data deleted by an administrator is also removed from your
disaster recovery backups within the retention period specified in the DPA. Could you
confirm this for us by 11:00 Monday?
If we can’t get legal comfortable with this point before then, we’ll need to push the
renewal signature into our next approval window.
metrics · last 24h
Customer
Requests
Errors
p95
7d
Acme
482,140
8
312ms
+3%
Contoso
821,334
22
284ms
+7%
Umbrella
94,217
163
1,821ms
−41%
Stark Industries
1,482,012
74
298ms
+9%
// the answer key
What the model needed to figure out
Five judgements decide most of the score. They are published with the scenario, so you can decide for yourself whether the benchmark is asking the right things.
threadwhat the sources saycorrect conclusion
01
Contoso
€72k ARR renewal.
Legal blocker.
Written technical confirmation required before 11:00.
Answer the data-retention question: confirm with the technical owner whether admin-deleted data is purged from disaster-recovery backups, and reply before 11:00.
02
Acme
Task list still says urgent.
The incident is already resolved.
Do not treat it as an active incident.
03
Analytics v2
Task list says Monday launch.
Latest information says Wednesday.
The launch email is owned by Julia.
Don’t execute the stale launch tasks.
04
Umbrella
Nobody mentions it in Slack or email.
The anomaly only exists in the metrics.
Investigate it today, but don’t invent a cause.
05
Churn
Enterprise churn rose from 3.1% to 4.4%.
There are known cancellations but insufficient evidence to explain the full increase.
Investigate the breakdown before Tuesday’s board meeting without claiming causality.
03
The method
How the score is built, and how to check it.
// scoring
How Monday Score works
One hundred points across 6 dimensions, then penalties. Quality only: speed and cost are reported separately, beside the score.
252515151010
Action detectionDid it find the work that had to happen today?25
State reconstructionDoes it know what is currently true?25
PrioritizationIs the order right, and is nothing stale promoted?15
Data reasoningDid it read the numbers, and read them correctly?15
UncertaintyDoes it separate what it knows from what it assumes?10
Noise resistanceDoes it stay away from work that is already done?10
total100
The judge does not assign a 0–100 score directly
It classifies atomic criteria one at a time, 19 of them in rubric 0.2, and each verdict carries the evidence it was based on. Deterministic scoring code turns those verdicts into points. No model ever sees a running total.
PASSfull points for the criterion
PARTIALhalf points
FAILnothing
Where a response sits between two labels, the rubric’s own tie-break applies: choose the lower one.
Penalties
Applied on top of the dimension score, capped at −20 per run.
−5Material hallucinationA claim that matters to the briefing and is not supported by the material.
−10Prohibited actionRecommending something the current state explicitly rules out.
−5Major contradictionContradicting itself about the state of an important issue.
Each scenario’s rubric sets its own penalty types, their prices and the cap. A prohibited action costs −10 (#001), −10 (#002), −7 (#003), −7 (#004), −7 (#005); the cap per run is −20 (#001), −30 (#002), −25 (#003), −25 (#004), −14 (#005).
Every input, raw response, judge verdict and scoring rule for this scenario is published. The hash below is of the assembled prompt each model received, byte for byte.
Benchmarks are useful, but what matters is your workload
We compare open models in production so you can tell how close they are to closed models. MondayBench is how we check whether they are really useful before they get there.