calendars.csv5 rows Who is free, and when · decides the release
accounts.csv6 rows Usage, admin logins, open tickets
contracts.csv6 rows Renewal windows and commercial context
support.csv6 rows Six open tickets
01
The result
Fifteen models, 45 runs, one judge.
// results
The leaderboard
Monday Score is quality only: how well each model answered “Can it tell what changed?”, criterion by criterion against this scenario’s rubric. Speed sits beside it and is never folded in.
All modelsOpen-weight models only
monday-002 · 15 models, 3 runs each
#
Model
Monday Score
Run range
TTFA median
Total latency
Show detail for
01
GPT-6 Astraclosed
100.0
0.0
—
—
Individual runs
Run 1100.0
Run 2100.0
Run 3100.0
Speed
TTFA
—
Total latency
—
Median output tokens
—
Dimension breakdown
State reconstruction 15/15
Temporal supersession 15/15
Planning and dependencies 25/25
Decision and prioritisation 20/20
Cross-source reasoning 15/15
Uncertainty 10/10
What it missed
Full marks on every criterion.
What it got wrong
No penalties and no false actions in any run.
02
Claude Opus 5closed
98.0
3.0
—
—
Individual runs
Run 199.0
Run 296.0
Run 399.0
Speed
TTFA
—
Total latency
—
Median output tokens
6,580
Dimension breakdown
State reconstruction 15/15
Temporal supersession 15/15
Planning and dependencies 24/25
Decision and prioritisation 20/20
Cross-source reasoning 15/15
Uncertainty 9/10
What it missed
Judge’s wording below, unedited.
No invented causes−1 pts forgone
no_unsupported_causality
R1 PARTIALR2 PARTIALR3 PARTIAL
Treats Globex support work as a path to 'unblock procurement' while the sources do not establish that causal link; elsewhere it recognises the uncertainty. Explicit hypothesis => PARTIAL.
Schedule actually fits−1 pts forgone
schedule_feasibility
R1 PASSR2 PARTIALR3 PASS
Its table gives Priya 09:12–09:35 for a 25-minute delta review (23 minutes), so the intended B chain is slightly inconsistent even though the rest of the schedule fits.
What it got wrong
No penalties and no false actions in any run.
0103
DeepSeek V4.1 Flash (1017)
95.2
8.0
84.3s
93.2s
Individual runs
Run 192.0
Run 293.5
Run 3100.0
Speed
TTFA
84.3s
Total latency
93.2s
Median output tokens
18,094
Dimension breakdown
State reconstruction 15/15
Temporal supersession 15/15
Planning and dependencies 22.3/25
Decision and prioritisation 18.3/20
Cross-source reasoning 15/15
Uncertainty 9.5/10
What it missed
Judge’s wording below, unedited.
Picks the safer release path−2.7 pts forgone
choose_option_b
R1 PARTIALR2 PARTIALR3 PASS
'Choose 2.9.5-patch (Option B) as the primary path'; the response never states B's <1% vs A's 8% session-reset risk or any reason for the choice.
Chooses B and schedules it viably but gives no risk-based rationale.
Globex elevated on all signals−1.7 pts forgone
globex_priority
R1 PARTIALR2 PARTIALR3 PASS
'Globex: highest customer/revenue support priority. €84k ARR, renewal in 18 days, €12k ... pending procurement, and three open tickets including high-severity SSO'; usage/admin decline not cited.
Elevates Globex but omits the usage/admin evidence the PASS bar names; between labels, so the lower one.
No time promised to the customer−0.5 pts forgone
conditional_launch_timing
R1 PARTIALR2 PASSR3 PASS
'09:45: Send Atlas a status update ... target rollout open by 11:00, will confirm when live', sent before review and deploy have finished.
Avoids a firm promise but communicates a time externally before the gated chain completes, so conditionality is insufficient.
What it got wrong
No penalties and no false actions in any run.
0204
Qwen 3.8 Flash
94.8
8.5
258.9s
285.0s
Individual runs
Run 193.0
Run 2100.0
Run 391.5
Speed
TTFA
258.9s
Total latency
285.0s
Median output tokens
33,714
Dimension breakdown
State reconstruction 15/15
Temporal supersession 15/15
Planning and dependencies 20.3/25
Decision and prioritisation 20/20
Cross-source reasoning 15/15
Uncertainty 9.5/10
What it missed
Judge’s wording below, unedited.
Picks the safer release path−2.7 pts forgone
choose_option_b
R1 PARTIALR2 PASSR3 PARTIAL
chooses viable A and schedules it correctly; B is set aside because 'there is no evidence the 2.9.5 diff is already submitted', which the response itself files as a thing to verify rather than a settled constraint
Schedule actually fits−2 pts forgone
schedule_feasibility
R1 PARTIALR2 PASSR3 PARTIAL
every constraint is named and the A sequence is exact, but no B schedule is committed - it is deferred to MUST VERIFY
No time promised to the customer−0.5 pts forgone
conditional_launch_timing
R1 PASSR2 PASSR3 PARTIAL
'Do not confirm Atlas launch at 10:30' is right, but '09:40-09:45 ... confirm to Atlas that rollout is scheduled to begin at 11:00' still goes out before Priya's verification, with the conditionality only implied by the preceding step
What it got wrong
No penalties and no false actions in any run.
0305
DeepSeek V4.1 Flash
93.0
10.5
61.2s
67.8s
Individual runs
Run 189.5
Run 2100.0
Run 389.5
Speed
TTFA
61.2s
Total latency
67.8s
Median output tokens
13,799
Dimension breakdown
State reconstruction 15/15
Temporal supersession 15/15
Planning and dependencies 18/25
Decision and prioritisation 20/20
Cross-source reasoning 15/15
Uncertainty 10/10
What it missed
Judge’s wording below, unedited.
Picks the safer release path−2.7 pts forgone
choose_option_b
R1 PARTIALR2 PASSR3 PARTIAL
The response rejects Option B by anchoring the delta-review start to Marcus's availability, an unsupported dependency.
Option A is viable, but the lower-risk Option B is incorrectly treated as non-viable.
Dependency chain in order−2.3 pts forgone
option_b_sequence
R1 PARTIALR2 PASSR3 PARTIAL
The required review-before-deploy rule is understood, but a complete viable Option B chain is not produced.
The model knows the dependency but does not execute the correct Option B sequence.
Schedule actually fits−2 pts forgone
schedule_feasibility
R1 PARTIALR2 PASSR3 PARTIAL
The chosen Option A schedule is feasible, but Option B feasibility is computed using an unsupported 09:15 start constraint.
The end-to-end feasibility check is only partly correct.
What it got wrong
No penalties and no false actions in any run.
0406
GLM 5.2
91.2
8.0
67.0s
85.4s
Individual runs
Run 190.5
Run 287.5
Run 395.5
Speed
TTFA
67.0s
Total latency
85.4s
Median output tokens
6,777
Dimension breakdown
State reconstruction 15/15
Temporal supersession 12.5/15
Planning and dependencies 18.7/25
Decision and prioritisation 20/20
Cross-source reasoning 15/15
Uncertainty 10/10
What it missed
Judge’s wording below, unedited.
Schedule actually fits−3 pts forgone
schedule_feasibility
R1 PARTIALR2 FAILR3 PASS
'Option B cannot fit Marcus's 09:15-10:20 window after a required 25-min delta review (review + 45-min deploy > 65 min available)' - the review runs before Marcus arrives, so the deploy has 09:32-10:17
Picks the safer release path−2.7 pts forgone
choose_option_b
R1 PARTIALR2 PARTIALR3 PASS
chooses A, which is viable and correctly scheduled; B is set aside because 'the 2.9.5 diff is already submitted' is 'not confirmed' - a defensible reading, but submitting it is an action the user could take at 09:07
Send handed to Daniel−2.5 pts forgone
ownership_supersession
R1 PARTIALR2 PARTIALR3 PARTIAL
names Daniel as the current sender and gates it on Sam's confirmation, but Sarah's handoff is never traced and the stale 'You' task is not displaced
Uses the waiting time−0.7 pts forgone
parallelize_work
R1 PASSR2 PASSR3 PARTIAL
the board pack and Globex triage are scheduled for the right day and the right deadline, but the plan does not say they run inside the delegated deploy window
What it got wrong
No penalties and no false actions in any run.
07
Claude Fable 5.1closed
91.0
6.0
—
—
Individual runs
Run 194.5
Run 288.5
Run 390.0
Speed
TTFA
—
Total latency
—
Median output tokens
7,226
Dimension breakdown
State reconstruction 15/15
Temporal supersession 15/15
Planning and dependencies 17/25
Decision and prioritisation 20/20
Cross-source reasoning 15/15
Uncertainty 9/10
What it missed
Judge’s wording below, unedited.
Picks the safer release path−4 pts forgone
choose_option_b
R1 PARTIALR2 PARTIALR3 PARTIAL
'Path chosen: Option A (2.9.4 + shard precheck), with B pre-reviewed as fallback'
chooses viable A correctly rather than B
Schedule actually fits−4 pts forgone
schedule_feasibility
R1 PASSR2 FAILR3 FAIL
'B requires a 25-min delta review to finish by 09:35 (Priya's cutoff) so that a 45-min deploy can finish by 10:20 (Marcus's cutoff) - zero slack at both ends, and Marcus isn't even online until 09:15' then 'Do not choose Option B - infeasible under today's availability'
POST-AUDIT CORRECTION PARTIAL -> FAIL (experiments/fable-5.1-full-mondaybench/AUDIT.md section 5). Original reason: states the constraint windows correctly - and both are satisfiable, review 09:07-09:32 then deploy 09:32-10:17 - but commits no B schedule and concludes infeasibility that its own windows do not support
No time promised to the customer−1 pts forgone
conditional_launch_timing
R1 PARTIALR2 PARTIALR3 PASS
section 4 'don't promise any time until Sam confirms rollout is open' against the 09:35-10:25 step 'Via Laura, reset Atlas expectation from 10:30 to migration starts ~11:00-11:15'
no firm promise, but an external time goes out before verification and Sam's confirmation
What it got wrong
No penalties and no false actions in any run.
0508
DeepSeek V4 Flash
89.3
13.5
162.6s
178.8s
Individual runs
Run 197.5
Run 286.5
Run 384.0
Speed
TTFA
162.6s
Total latency
178.8s
Median output tokens
14,863
Dimension breakdown
State reconstruction 15/15
Temporal supersession 13.3/15
Planning and dependencies 16/25
Decision and prioritisation 20/20
Cross-source reasoning 15/15
Uncertainty 10/10
What it missed
Judge’s wording below, unedited.
Schedule actually fits−4 pts forgone
schedule_feasibility
R1 PASSR2 FAILR3 FAIL
no B schedule is produced; 'B cannot meet the 12:00 Atlas window if started after 09:35' is true as stated but the 09:07 start is never considered
Picks the safer release path−2.7 pts forgone
choose_option_b
R1 PASSR2 PARTIALR3 PARTIAL
chooses viable A and schedules it correctly, on a false claim about B
Dependency chain in order−2.3 pts forgone
option_b_sequence
R1 PASSR2 PARTIALR3 PARTIAL
'Do not use 2.9.5-patch without a completed Security delta review' places the review before deploy, but the B chain is not carried through verification and Sam
Send handed to Daniel−1.7 pts forgone
ownership_supersession
R1 PARTIALR2 PASSR3 PARTIAL
names Daniel as the current sender and gates it on Sam's confirmation, but Sarah's handoff is never traced and the stale 'You' task is not displaced
What it got wrong
No penalties and no false actions in any run.
0609
GLM 5.3
87.7
4.0
176.2s
197.1s
Individual runs
Run 185.5
Run 288.0
Run 389.5
Speed
TTFA
176.2s
Total latency
197.1s
Median output tokens
10,603
Dimension breakdown
State reconstruction 15/15
Temporal supersession 15/15
Planning and dependencies 14.5/25
Decision and prioritisation 20/20
Cross-source reasoning 15/15
Uncertainty 8.2/10
What it missed
Judge’s wording below, unedited.
Picks the safer release path−4 pts forgone
choose_option_b
R1 PARTIALR2 PARTIALR3 PARTIAL
chooses A, which is time-viable, and schedules it correctly - but on the false ground that 'B cannot physically complete today'. The 25-minute delta review can run 09:07-09:32 inside Priya's first window, leaving Marcus 09:32-10:17 for the 45-minute deploy. A's 8% session-reset risk is accepted where B's is under 1%
Dependency chain in order−3.5 pts forgone
option_b_sequence
R1 PARTIALR2 PARTIALR3 PARTIAL
no B sequence is produced, but the gating dependency is stated - 'never deploy it before Priya's delta review' - and the rest of the chain (deploy, post-deploy verification, Sam rollout) is ordered correctly for the path chosen
Schedule actually fits−3 pts forgone
schedule_feasibility
R1 PARTIALR2 PARTIALR3 PARTIAL
the A schedule is coherent and fits Marcus 09:15-09:35, Priya 10:25-10:45, Sam 10:45-11:00 and the before-12 window; no feasible B schedule is produced and B's feasibility is computed wrongly
No time promised to the customer−1.5 pts forgone
conditional_launch_timing
R1 FAILR2 PARTIALR3 PASS
'~09:40 - Confirm to Atlas (via Laura): launch is on today; rollout opens ~10:45-11:00, inside their window' - a firm external commitment to a rollout window, issued before Priya's verification and Sam's rollout have happened, with no conditionality anywhere in the response. PARTIAL is for a non-firm promise with thin conditionality; this is a firm one
No invented causes−0.3 pts forgone
no_unsupported_causality
R1 PARTIALR2 PASSR3 PASS
'Soylent: -22%/-28% plus a medium ticket citing slower onboarding. Possible link (hypothesis only)' - an unsupported cause offered, but explicitly as a hypothesis, which is the PARTIAL band exactly
What it got wrong
No penalties and no false actions in any run.
0710
GLM 5.3 Flash
87.3
6.5
238.5s
316.6s
Individual runs
Run 185.0
Run 285.5
Run 391.5
Speed
TTFA
238.5s
Total latency
316.6s
Median output tokens
12,394
Dimension breakdown
State reconstruction 15/15
Temporal supersession 15/15
Planning and dependencies 13.7/25
Decision and prioritisation 20/20
Cross-source reasoning 15/15
Uncertainty 8.7/10
What it missed
Judge’s wording below, unedited.
Schedule actually fits−5 pts forgone
schedule_feasibility
R1 FAILR2 FAILR3 PARTIAL
'Option B needs the delta review done by ~09:35 and a 45-min deploy finished before Marcus leaves at 10:20 - zero slack anywhere'; the review can start at 09:07 and the deploy ends 10:17
Picks the safer release path−4 pts forgone
choose_option_b
R1 PARTIALR2 PARTIALR3 PARTIAL
chooses viable A and schedules it correctly, but 'B is not viable today' overstates: the review runs in Priya's 09:00-09:35 window and the deploy ends 10:17
Dependency chain in order−2.3 pts forgone
option_b_sequence
R1 PARTIALR2 PARTIALR3 PASS
'B needs a 25-min delta review first' and 'Don't deploy 2.9.5-patch before Priya's delta review completes' place the gate correctly, but verification and Sam are not carried into the B chain
No time promised to the customer−1 pts forgone
conditional_launch_timing
R1 PARTIALR2 PASSR3 PARTIAL
kills the 10:30 task, but 'By 09:45 - Confirm to Atlas: launch proceeds today, rollout opens ~11:00' goes out before the deploy has been verified
No invented causes−0.3 pts forgone
no_unsupported_causality
R1 PASSR2 PARTIALR3 PASS
'The -61% admin-logins drop is consistent with the SSO breakage - actionable today via the ticket, not a mystery' presents the link as settled rather than as a hypothesis
What it got wrong
No penalties and no false actions in any run.
0811
MiMo v2.6 Flash
83.7
4.0
257.0s
322.0s
Individual runs
Run 182.0
Run 283.0
Run 386.0
Speed
TTFA
257.0s
Total latency
322.0s
Median output tokens
10,625
Dimension breakdown
State reconstruction 14.2/15
Temporal supersession 14.2/15
Planning and dependencies 14.5/25
Decision and prioritisation 19.2/20
Cross-source reasoning 15/15
Uncertainty 6.7/10
What it missed
Judge’s wording below, unedited.
Picks the safer release path−4 pts forgone
choose_option_b
R1 PARTIALR2 PARTIALR3 PARTIAL
'B cannot be sequenced before Marcus leaves → A is the viable path'; would switch to B only if diff already submitted by 09:10.
Chooses viable Option A; does not choose B (and wrongly rules it infeasible).
Dependency chain in order−3.5 pts forgone
option_b_sequence
R1 PARTIALR2 PARTIALR3 PARTIAL
'B requires Priya's 25-min delta review to finish by 09:35 so Marcus can deploy'; 'Don't deploy 2.9.5 ... without Priya's 25-min delta review'; plan follows deploy -> 10:25 verification -> Sam 10:45.
Understands the B dependencies but never sequences delta review -> B deploy -> verification -> rollout as the plan.
Schedule actually fits−3 pts forgone
schedule_feasibility
R1 PARTIALR2 PARTIALR3 PARTIAL
A schedule: deploy 09:15-09:35, Priya 10:25-10:45, Sam 10:45-11:00, migration before 12:00; no B schedule.
Major constraints understood and timed coherently, but no coherent B schedule.
No time promised to the customer−2 pts forgone
conditional_launch_timing
R1 PARTIALR2 FAILR3 PARTIAL
'09:45 Confirm revised launch timeline to Atlas: deploy at 09:15, post-deploy verification starts 10:25, rollout opens ~10:45'.
Commits a rollout time externally to Atlas without conditioning it on deploy/verification success.
No invented causes−1.3 pts forgone
no_unsupported_causality
R1 PARTIALR2 PARTIALR3 FAIL
'The −61% is not a mystery metric: it corroborates the SSO blocker.'
Asserts an unsupported cause for Globex's admin-login drop as fact, not as a labelled hypothesis.
Pricing authority resolved−0.8 pts forgone
pricing_authority
R1 PARTIALR2 PASSR3 PASS
'nothing external'; but 'The pinned v4-final.xlsx (13:32 Friday) landed after Daniel's 09:14 "not authoritative" email — internal precedence unresolved'; Unclear #5 asks which internal source governs.
Rejects external use but leaves authority between Daniel's values and the pinned workbook unresolved.
Globex elevated on all signals−0.8 pts forgone
globex_priority
R1 PARTIALR2 PASSR3 PASS
'Globex — top priority. 18-day renewal + €12k expansion pending procurement + 3 open tickets incl. high-severity G-441'; no mention of Globex usage −18% or admin logins −61%.
Elevates Globex but omits the usage/admin signal from its rationale.
Stale external actions refused−0.8 pts forgone
stale_external_actions
R1 PASSR2 PARTIALR3 PASS
Do-not list rejects 'send Enterprise pricing to Initech procurement' and 'call Zenith about renewal risk'; the announcement is assigned to Daniel in the plan but the stale 'You send announcement' task is never explicitly rejected.
Clearly rejects two of the three stale external actions; user launch-send is only implicitly avoided, so lower label.
What it got wrong
No penalties and no false actions in any run.
0912
MIMO v2.5
81.2
33.0
163.3s
195.4s
Individual runs
Run 164.5
Run 281.5
Run 397.5
Speed
TTFA
163.3s
Total latency
195.4s
Median output tokens
9,510
Dimension breakdown
State reconstruction 14.2/15
Temporal supersession 12.5/15
Planning and dependencies 14.8/25
Decision and prioritisation 18.3/20
Cross-source reasoning 14/15
Uncertainty 9/10
What it missed
Judge’s wording below, unedited.
Schedule actually fits−4 pts forgone
schedule_feasibility
R1 FAILR2 FAILR3 PASS
'with Priya unavailable 09:35-10:25 and Marcus departing 10:20, the windows don't overlap enough to complete delta review -> 45-min deploy -> post-deploy verification before 12:00' - Priya is available 09:00-09:35, so the review runs 09:07-09:32
Dependency chain in order−3.5 pts forgone
option_b_sequence
R1 FAILR2 PARTIALR3 PASS
no B chain is given; the contingency - 'fall to Option B with a delta review Monday afternoon' - is offered for a moment when Marcus, the only deployer, has already left
Picks the safer release path−2.7 pts forgone
choose_option_b
R1 PARTIALR2 PARTIALR3 PASS
chooses viable A; the rationale is 'fewer moving parts' and an accepted 8% risk rather than a comparison the evidence supports
Send handed to Daniel−2.5 pts forgone
ownership_supersession
R1 PARTIALR2 PARTIALR3 PARTIAL
names Daniel as the current sender and gates it on Sam's confirmation, but Sarah's handoff is never traced and the stale 'You' task is not displaced
Globex elevated on all signals−1.7 pts forgone
globex_priority
R1 PARTIALR2 PARTIALR3 PASS
'3 open tickets, one high-severity (SSO blocker). September expansion (EUR12k) stalled in procurement. Renewal in 18 days' - the usage -18% and admin -61% signals are never brought in
€54k gap closed−1 pts forgone
expansion_fully_reconciled
R1 PARTIALR2 PASSR3 PASS
Wayne EUR18k, Globex EUR12k, Soylent EUR10k and Wonka EUR14k all appear, and 'Together these are the entire EUR24k unexplained gap', but the four are never closed against the EUR54k
No time promised to the customer−1 pts forgone
conditional_launch_timing
R1 FAILR2 PASSR3 PASS
'09:45 | Confirm to Atlas: Enterprise Controls launch is on for today; admins should be standing by. Migration target ~11:00' - an unconditional external commitment made before the deploy is verified
Release state is current−0.8 pts forgone
launch_state
R1 PARTIALR2 PASSR3 PASS
the recovery state and Maya's removed gate are right, but '09:45 | Confirm to Atlas: Enterprise Controls launch is on for today... Migration target ~11:00' is sent before any verification
What it got wrong
MAJOR_CONTRADICTION-5 · Run 1
'09:35 | Deploy complete. No post-deploy verification needed (no code change from staging-verified build).' against '10:25-10:45 | Priya runs post-deploy verification (~20 min)'
The timeline states that Security's post-deploy check is not required and then schedules it. Priya's policy is explicit that a verified build still needs a production post-deploy check, so the first statement is also wrong on the policy - a reader following the timeline in order would skip a mandatory gate.
13
Gemini 3.8 Flash (High)closed
80.0
1.5
—
38.8s (route-timed)
Individual runs
Run 181.0
Run 279.5
Run 379.5
Speed
TTFA
—
Total latency
38.8s · route-timed
Median output tokens
—
Dimension breakdown
State reconstruction 15/15
Temporal supersession 15/15
Planning and dependencies 8/25
Decision and prioritisation 20/20
Cross-source reasoning 14.5/15
Uncertainty 7.5/10
What it missed
Judge’s wording below, unedited.
Dependency chain in order−7 pts forgone
option_b_sequence
R1 FAILR2 FAILR3 FAIL
No B sequence planned; "DO NOT deploy Option B (Build 2.9.5-patch)".
The B dependency chain is not planned.
Schedule actually fits−6 pts forgone
schedule_feasibility
R1 FAILR2 FAILR3 FAIL
Asserts "Priya cannot complete a 25-minute review before 09:35" and Marcus has "no window" for the 45-minute deploy; no B schedule.
No coherent B schedule; it misreads the timing that makes B fit.
Picks the safer release path−4 pts forgone
choose_option_b
R1 PARTIALR2 PARTIALR3 PARTIAL
Chooses "Option A (Build 2.9.4 with shard precheck)" with a viable 09:15-09:35 deploy, 10:25-10:45 verification, 10:45-11:00 rollout; but claims B "cannot complete deployment before Marcus departs" (70 min vs 65 min).
Chooses a viable A with a working schedule, which is the PARTIAL bar; does not choose B, and its stated reason B is infeasible ignores the async delta review Priya can start before 09:35.
No invented causes−2 pts forgone
no_unsupported_causality
R1 FAILR2 FAILR3 FAIL
Globex: "G-441 ... shows admins cannot complete SSO setup, directly driving the 61% login drop" stated as fact.
Asserts an unsupported customer cause as fact, not as a hypothesis.
Arithmetic kept apart from cause−0.5 pts forgone
causal_boundary
R1 PASSR2 PARTIALR3 PASS
Reconciliation keeps Soylent/Wonka as "contract has no stated cause", but the priorities section asserts "Onboarding friction is depressing license adoption" for Soylent and "Classic silent disengagement" for Wonka.
Cautious in the reconciliation but blurs into stated causes for Soylent/Wonka elsewhere.
No time promised to the customer−0.5 pts forgone
conditional_launch_timing
R1 PASSR2 PASSR3 PARTIAL
"Instruct Laura (CS) not to promise a 10:30 launch to Atlas, but reassure them rollout will open well before the 12:00 deadline."
Avoids the 10:30 promise but tells CS to give Atlas an unconditional external assurance about the rollout time.
What it got wrong
No penalties and no false actions in any run.
1014
Qwen 3.6
70.2
31.0
0.9s
18.2s
Individual runs
Run 175.5
Run 283.0
Run 352.0
Speed
TTFA
0.9s
Total latency
18.2s
Median output tokens
2,644
Dimension breakdown
State reconstruction 12.5/15
Temporal supersession 11.7/15
Planning and dependencies 14.2/25
Decision and prioritisation 16.8/20
Cross-source reasoning 13.5/15
Uncertainty 7.8/10
What it missed
Judge’s wording below, unedited.
Schedule actually fits−5 pts forgone
schedule_feasibility
R1 FAILR2 PARTIALR3 FAIL
'Priya verifies 09:35-09:55 (after her break)' misreads calendars.csv - 09:35-10:25 is the break - and the whole A timeline, 'Rollout opens ~10:15. Feasible', is built on it
Picks the safer release path−4 pts forgone
choose_option_b
R1 PARTIALR2 PARTIALR3 PARTIAL
chooses A, which is viable, but calls it 'the only viable path' while its own contingency shows B is 'tight but possible'
Globex elevated on all signals−2.5 pts forgone
globex_priority
R1 PARTIALR2 PARTIALR3 PARTIAL
'3 open tickets (High/Medium/Medium). SSO setup failure (G-441) is a blocker for admin operations. Expansion is pending procurement' - the 18-day renewal and the usage/admin figures are absent
Release state is current−2.5 pts forgone
launch_state
R1 PASSR2 PARTIALR3 FAIL
Maya's removal of the executive gate never appears; the stale 09:15 approval task is left standing; and '#leadership: "Will confirm rollout open by 10:30"' revives the stale launch time
CEO gate is gone−1.7 pts forgone
executive_gate_superseded
R1 PASSR2 PARTIALR3 PARTIAL
no approval is sought, but the supersession is never established and the stale task is not rejected
Send handed to Daniel−1.7 pts forgone
ownership_supersession
R1 PASSR2 PARTIALR3 PARTIAL
names Daniel as the current sender and gates it on Sam's confirmation, but Sarah's handoff is never traced and the stale 'You' task is not displaced
No time promised to the customer−1.5 pts forgone
conditional_launch_timing
R1 PARTIALR2 PASSR3 FAIL
'#leadership: "Maya, selecting Option A... Will confirm rollout open by 10:30"' - a commitment to a time the response's own impossible schedule produced
Dependency chain in order−1.2 pts forgone
option_b_sequence
R1 PARTIALR2 PASSR3 PASS
'Option B: Marcus deploy -> Priya 25m delta review (async) -> Priya 20m post-deploy verify -> Sam 15m rollout check' puts the delta review after the deploy
€54k gap closed−1 pts forgone
expansion_fully_reconciled
R1 PASSR2 PASSR3 PARTIAL
'Explained (EUR30k): Wayne EUR18k; Globex EUR12k' and 'Remaining Gap (EUR24k)' are right, but the response concludes 'exact arithmetic requires full account file review' and never closes Soylent EUR10k + Wonka EUR14k
Uses the waiting time−0.7 pts forgone
parallelize_work
R1 PASSR2 PASSR3 PARTIAL
'10:30-11:00 Board Pack Expansion Reconciliation' sits after the release rather than inside the delegated waiting window
Acme is not churn−0.7 pts forgone
avoid_acme_false_positive
R1 PASSR2 PASSR3 PARTIAL
'Acme: Medium Risk. Structural shift to annual batch processing. Usage down 45%... Monitor for churn' - the explanation is noticed and then the account is still carried as a churn watch
No invented causes−0.7 pts forgone
no_unsupported_causality
R1 PASSR2 PASSR3 FAIL
'The remaining EUR24k gap is attributed to Soylent and Wonka usage declines and Acme's structural shift' asserts a cause Maya explicitly forbade inventing, and Acme's expansion is unchanged at zero
Arithmetic kept apart from cause−0.5 pts forgone
causal_boundary
R1 PASSR2 PASSR3 PARTIAL
'Do NOT invent causes for the EUR24k expansion gap' is stated and then broken in the same section by 'attributed to Soylent and Wonka usage declines and Acme's structural shift'
What it got wrong
MAJOR_PLANNING_ERROR-7 · Run 1
'Monitor Post-Deploy Verify: Priya performs 20-minute verification (09:35-09:55)' and 'If verify finishes at 09:55, rollout opens ~10:10'
calendars.csv and Priya's own email both put her out of contact 09:35-10:25. The recommended plan books her required verification inside that window and derives a rollout time from it, so the schedule the response tells the user to execute cannot run.
MAJOR_PLANNING_ERROR-7 · Run 3
'Priya verifies 09:35-09:55 (after her break). Sam checks 09:55-10:10. Rollout opens ~10:15. Feasible.'
Priya's break is 09:35-10:25, not before it. The recommended plan books her mandatory verification inside her stated unavailable window and then commits a rollout time to the CEO on that basis, so the plan the response tells the user to execute cannot run.
MATERIAL_HALLUCINATION-5 · Run 3
'The remaining EUR24k gap is attributed to Soylent and Wonka usage declines and Acme's structural shift'
No source establishes usage decline as the cause of either expansion reduction, and Acme's September expansion is unchanged at zero so it contributes nothing to the gap. Maya's brief specifically forbids supplying causes the data does not establish, and this goes into the board pack.
1115
Gemma 4
54.2
13.5
0.3s
8.7s
Individual runs
Run 158.0
Run 245.5
Run 359.0
Speed
TTFA
0.3s
Total latency
8.7s
Median output tokens
921
Dimension breakdown
State reconstruction 10/15
Temporal supersession 9.2/15
Planning and dependencies 12/25
Decision and prioritisation 15.8/20
Cross-source reasoning 9/15
Uncertainty 6.8/10
What it missed
Judge’s wording below, unedited.
Schedule actually fits−6 pts forgone
schedule_feasibility
R1 FAILR2 FAILR3 FAIL
'10:25 - 11:15: Deployment Sequence: Marcus deploys Option B (45m) -> Priya (20m) -> Sam (15m)' - Marcus is gone at 10:20, and 80 minutes of work is booked into a 50-minute slot
Release state is current−5 pts forgone
launch_state
R1 FAILR2 FAILR3 FAIL
Maya's removal of the executive gate never appears, and 'Notify Atlas of the 10:30 launch target' revives the stale launch time
Picks the safer release path−4 pts forgone
choose_option_b
R1 PARTIALR2 PARTIALR3 PARTIAL
chooses B for the right reason - 'to minimize session reset risk for Atlas' - but never shows it viable, and the schedule it then writes is not
CEO gate is gone−3.3 pts forgone
executive_gate_superseded
R1 PARTIALR2 FAILR3 PARTIAL
'Confirm Enterprise Controls launch path with Maya' seeks exactly the approval Maya said she would not add
€54k gap closed−3 pts forgone
expansion_fully_reconciled
R1 PARTIALR2 PARTIALR3 PARTIAL
'EUR30k of the gap is explained by Wayne and Globex. The remaining EUR24k is driven by account-level reductions (Soylent, Wonka, Acme)' - the split is right but Acme's expansion is unchanged at zero, so the EUR24k attribution never closes
Three files read together−3 pts forgone
customer_signal_integration
R1 PARTIALR2 PARTIALR3 PARTIAL
Globex and Acme are each crossed against two sources, but Soylent and Wonka never reach the customer priorities at all
Send handed to Daniel−2.5 pts forgone
ownership_supersession
R1 PARTIALR2 PARTIALR3 PARTIAL
names Daniel as the current sender and gates it on Sam's confirmation, but Sarah's handoff is never traced and the stale 'You' task is not displaced
Globex elevated on all signals−2.5 pts forgone
globex_priority
R1 PARTIALR2 PARTIALR3 PARTIAL
'Active contract (18 days to renewal) with high-severity SSO/SCIM support tickets and a pending EUR12k expansion' - three of the four signals, with usage -18% and admin -61% missing
No time promised to the customer−2.5 pts forgone
conditional_launch_timing
R1 FAILR2 PARTIALR3 FAIL
'Notify Atlas of the 10:30 launch target' at 09:30-09:45 - an unhedged external commitment to a time the response's own timeline opens at 11:30-12:00
Dependency chain in order−2.3 pts forgone
option_b_sequence
R1 PARTIALR2 PARTIALR3 PASS
'Marcus deploys 2.9.5-patch (45m). Priya performs 25m delta review (asynchronous) + 20m post-deploy verification' puts the mandatory pre-deploy review after the deploy, contradicting the 09:15 instruction to submit the diff first
Board pack before 11:00−1.7 pts forgone
expansion_before_11
R1 PASSR2 PARTIALR3 PARTIAL
'11:00: Submit expansion reconciliation to Maya' lands on the deadline itself, after the 10:45-11:00 rollout step, with no earlier draft
Uses the waiting time−0.7 pts forgone
parallelize_work
R1 PASSR2 PARTIALR3 PASS
the board pack is submitted at 11:00 as a discrete step rather than drafted inside the delegated deploy window
No invented causes−0.7 pts forgone
no_unsupported_causality
R1 PASSR2 FAILR3 PASS
'3 open support tickets (including high-severity SSO issue) impacting the pending September expansion' asserts a causal link between the tickets and the procurement delay that no source establishes
What it got wrong
MAJOR_PLANNING_ERROR-7 · Run 1
'10:30 - 11:30: Execution Sequence: Marcus deploys 2.9.5-patch (45m)'
calendars.csv and Marcus's own email both put his hard stop at 10:20. The plan schedules his 45-minute deploy to start ten minutes after he leaves, so the release the response recommends cannot be executed at all.
MAJOR_PLANNING_ERROR-7 · Run 2
'09:45 - 10:15: Coordinate with Marcus to initiate deployment of Option B. (Marcus must start by 09:15 and leave by 10:20; deployment takes 45m).'
The plan starts a 45-minute deploy at 09:45-10:15 while quoting, in the same line, that Marcus leaves at 10:20. The recommended release cannot complete under the availability the response itself cites.
MATERIAL_HALLUCINATION-5 · Run 2
'3 open support tickets (including high-severity SSO issue) impacting the pending September expansion'
contracts.csv states only that Globex's September expansion is pending procurement. Nothing links the support tickets to the procurement delay, and Maya's brief explicitly forbids supplying causes the data does not establish.
MAJOR_PLANNING_ERROR-7 · Run 3
'10:25 - 11:15: Deployment Sequence: Marcus deploys Option B (45m) -> Priya performs post-deploy verification (20m) -> Sam performs rollout sanity check (15m)'
Marcus's hard stop is 10:20, so the deploy has no deployer, and the three steps total 80 minutes inside a 50-minute block. The release the response recommends is unexecutable on both counts.
Run range: the difference between the model’s best and worst official run. It is a plain best-minus-worst span, so it is never written as ±.
TTFA, time to first answer: the wall-clock delay before the first visible answer token. Reasoning tokens do not count as answer content.
// quality vs speed
The smartest model isn’t always the fastest
Of the timed models, DeepSeek V4.1 Flash (1017) scores highest, 95.2, and takes 84.3s to start answering. Gemma 4 is the quickest: it starts in 0.3s and scores 54.2. Qwen 3.8 Flash is the slowest, 258.9s, for 94.8.
Monday Score and time to first answer, per model
Model
Monday Score
TTFA median
DeepSeek V4.1 Flash (1017)
95.2
84.3s
Qwen 3.8 Flash
94.8
258.9s
DeepSeek V4.1 Flash
93.0
61.2s
GLM 5.2
91.2
67.0s
DeepSeek V4 Flash
89.3
162.6s
GLM 5.3
87.7
176.2s
GLM 5.3 Flash
87.3
238.5s
MiMo v2.6 Flash
83.7
257.0s
MIMO v2.5
81.2
163.3s
Qwen 3.6
70.2
0.9s
Gemma 4
54.2
0.3s
// what stood out
Benchmark curiosities
Every figure below is one you can check against the leaderboard, the criterion matrix or the published verdicts on the same page.
01
12 / 45
runs chose the safer path
Option B
Most runs traded safety for a deadline they were not short of
Both recovery paths open rollout at 11:00, an hour inside the cutoff. Option B carries an eighth of the session-reset risk at no cost in time. Thirty-three of forty-five runs did not choose it. GPT-6 Astra and Claude Opus 5 chose it on all three of their runs; no open model chose it on more than one of three. The plan fits only if the sequence starts at 09:07, when the diff can be submitted; starting at 09:15, when the engineer arrives, leaves too little time.
02
#3 → #10
between #001 and #002
GLM 5.3 Flash
The best open model on the first scenario came tenth in this one
The two scenarios test different skills. #001 rewards noticing what changed; #002 rewards building a plan that runs. GLM 5.3 Flash’s three runs land within 6.5 points of each other, and none of them chose the safer release path.
03
64.5 – 97.5
three runs, same prompt
MIMO v2.5
One model produced its best and its worst answer on the same input
A 33-point range, the widest here. Its best run, taken alone, would place third on this board of fifteen; its worst, which states that Security verification is not needed and then schedules it anyway, would place fourteenth. Three runs per model exists for exactly this: on one run, MIMO looks like a top-three model or a bottom-two one depending on which you draw.
04
3 / 3
plans that cannot be executed
Gemma 4
One model recommended an impossible plan on every run
All three of its runs schedule the deploy to start after the only engineer who can run it has left — one of them at 10:30, ten minutes past his hard stop. That is what the MAJOR_PLANNING_ERROR penalty is scoped to, and Gemma 4 is the only model that triggered it on every one of its runs.
02
The test
What they were given, and what they had to work out.
// the test
This is a problem of joining scattered pieces
On Friday the production deploy of build 2.9.4 failed on shard eu-03 before rollout opened, and Atlas, a signed customer, needs that rollout to have begun by 12:00 Monday or its migration slips to Thursday. There are two ways back: Option A retries 2.9.4 with a shard precheck (8% session-reset risk); Option B ships 2.9.5-patch (under 1%), which, being a code change, first needs Priya’s 25-minute Security delta review. Either way she runs a 20-minute post-deploy verification before Sam can open rollout.
Who is free, and when
now · 09:07
Marcus leaves · 10:20
Atlas rollout must have begun · 12:00
The two recovery paths · A
2.9.4
Retry the build that already failed once, with a shard precheck.
deploy
20 min
session-reset risk
8%
no code change
The two recovery paths · B
2.9.5-patch
A patch that avoids the migration path that broke. Needs a Security delta review first.
deploy
45 min
session-reset risk
<1%
code change
Option B, sequenced
Option B’s deploy finishes 3 minutes before Marcus leaves, and that is all the margin it has. Rollout still opens at 11:00 on both paths, because both wait for Priya’s verification after she is back at 10:25.
The step that makes it fit
Priya reviews asynchronously from a submitted diff, and she is free from 09:00. The chain starts at 09:07 with an action the user can take, without waiting for Marcus to arrive at 09:15.
What happened
Both paths open rollout at 11:00, 60 minutes inside the deadline. Option B carries an eighth of the risk at no cost in time. 12 of 45 official runs chose it; 31 of 45 never built a schedule that showed it fits.
Miss it and migration slips to Thursday. The renewal is signed, so the cost is executive noise; revenue is not at stake.
// the answer key
What a right answer does
01
Choose Option B and submit the diff at 09:07. Review 09:07–09:32, deploy 09:32–10:17 (three minutes before Marcus’s 10:20 hard stop), verification 10:25–10:45 once Priya is back, rollout 10:45–11:00. Option A also opens rollout at 11:00, because it waits for the same verification: it saves no time and carries eight times the risk.
02
Close the €54k expansion gap before 11:00: Wayne €18k, Globex €12k, Soylent €10k, Wonka €14k. Only Wayne (a signed move to October) and Globex (procurement) have a stated cause.
03
Globex is the real account risk: renewal in 18 days, usage −18%, admin logins −61%, three open tickets. Acme’s −45% is the approved shift to batch processing that its contract notes, so it does not count as churn.
04
Don’t reinstate the CEO sign-off, promise a launch time to a customer, send the launch email yourself (Daniel owns it now) or send Enterprise pricing out from the stale internal workbook.
05
Orion stays open: neither ON nor OFF until someone confirms which org ID is theirs and reads the live flag for it.
One hundred points across 6 dimensions, then penalties. Quality only: speed and cost are reported separately, beside the score.
151525201510
State reconstructionDoes it know what is currently true?15
Temporal supersessionDoes it notice what a later message overruled?15
Planning and dependenciesCan it build a plan that would actually run?25
Decision and prioritisationDoes it act on the tightest constraint first?20
Cross-source reasoningDoes it combine files, and keep arithmetic apart from cause?15
UncertaintyDoes it refuse to settle what the evidence cannot?10
total100
The judge does not assign a 0–100 score directly
It classifies atomic criteria one at a time, 20 of them in rubric 0.2, and each verdict carries the evidence it was based on. Deterministic scoring code turns those verdicts into points. No model ever sees a running total.
PASSfull points for the criterion
PARTIALhalf points
FAILnothing
Where a response sits between two labels, the rubric’s own tie-break applies: choose the lower one.
Penalties
Applied on top of the dimension score, capped at −30 per run.
−5Material hallucinationA claim that matters to the briefing and is not supported by the material.
−10Prohibited actionRecommending something the current state explicitly rules out.
−7Major planning errorRecommending a plan that cannot run under the stated dependencies or availability.
−5Major contradictionContradicting itself about the state of an important issue.
Each scenario’s rubric sets its own penalty types, their prices and the cap. A prohibited action costs −10 (#001), −10 (#002), −7 (#003), −7 (#004), −7 (#005); the cap per run is −20 (#001), −30 (#002), −25 (#003), −25 (#004), −14 (#005).
Every input, raw response, judge verdict and scoring rule for this scenario is published. The hash below is of the assembled prompt each model received, byte for byte.
A run can be rejected before it is ever scored: truncated, or returning no reasoning when reasoning was requested, or a cached duplicate. Those never reach a leaderboard, because a leaderboard can only exclude what it evaluated. It describes the model and its endpoint, so it is reported apart from the score.
Qwen 3.8 Flash 3 of 4 attempts REASONING_NOT_DELIVERED
One run was generated and scored but does not count
monday-002__qwen3.8-flash__00266.5The provider served this run without the reasoning phase its two sibling runs used — zero reasoning tokens against 42,808 and 30,342 — so it was not comparable with them. Its score and its evidence stay published, and it is left out of the model’s result.
// why we built this
Benchmarks are useful, but what matters is your workload
We compare open models in production so you can tell how close they are to closed models. MondayBench is how we check whether they are really useful before they get there.