New: DeepSeek V4.1 Flash.The architecture built for agents, and the 890 bytes that change an agent bill.Explore the piece →
// mondaybench #003beta
We gave 15 models the same messy Monday morning
Six sources. Six dated commitments across one week, three of them waiting on the same three-hour migration, and one engineer claimed by two of them at once.
One question: Can it find the path through dependencies?
// six dated commitments · one week · 3 waits on the billing migration
now · Monday 09:10
waits on the billing migration
owes it nothing
01
The result
Fifteen models, 45 runs, one judge.
// results
The leaderboard
Monday Score is quality only: how well each model answered “Can it find the path through dependencies?”, criterion by criterion against this scenario’s rubric. Speed sits beside it and is never folded in.
All modelsOpen-weight models only
monday-003 · 15 models, 3 runs each
#
Model
Monday Score
Run range
TTFA median
Total latency
Show detail for
01
GPT-6 Astraclosed
100.0
0.0
—
—
Individual runs
Run 1100.0
Run 2100.0
Run 3100.0
Speed
TTFA
—
Total latency
—
Median output tokens
—
Dimension breakdown
Critical path and dependencies 21/21
Resource allocation 25/25
Planning and parallelisation 18/18
Decision quality 14/14
State reconstruction 10/10
Accuracy and uncertainty 12/12
What it missed
Full marks on every criterion.
What it got wrong
No penalties and no false actions in any run.
02
Claude Opus 5closed
98.3
5.0
—
—
Individual runs
Run 195.0
Run 2100.0
Run 3100.0
Speed
TTFA
—
Total latency
—
Median output tokens
6,474
Dimension breakdown
Critical path and dependencies 21/21
Resource allocation 25/25
Planning and parallelisation 18/18
Decision quality 14/14
State reconstruction 10/10
Accuracy and uncertainty 10.3/12
What it missed
Judge’s wording below, unedited.
Supplied values carried through−1.7 pts forgone
factual_plan_accuracy
R1 FAILR2 PASSR3 PASS
Uses multiple wrong factual-anchor deadlines: pricing_page is called 'deadline Wed 17:00' although the anchor is Thu 24 Sep 17:00, and onboarding_launch is called 'deadline Thu 12:00' although the anchor is Fri 25 Sep 12:00. PARTIAL allows one minor substitution; multiple substitutions fall to FAIL.
What it got wrong
No penalties and no false actions in any run.
03
Claude Fable 5.1closed
96.7
5.0
—
—
Individual runs
Run 1100.0
Run 295.0
Run 395.0
Speed
TTFA
—
Total latency
—
Median output tokens
6,713
Dimension breakdown
Critical path and dependencies 21/21
Resource allocation 25/25
Planning and parallelisation 18/18
Decision quality 14/14
State reconstruction 10/10
Accuracy and uncertainty 8.7/12
What it missed
Judge’s wording below, unedited.
Supplied values carried through−3.3 pts forgone
factual_plan_accuracy
R1 PASSR2 FAILR3 FAIL
onboarding launch given 'due Thu 12:00' where it is 2026-09-25T12:00 (Friday), and the board pack described as 'well before Thu 10:00 / Friday board' where the deadline is 2026-09-25T10:00 (Friday)
POST-AUDIT CORRECTION PARTIAL -> FAIL (experiments/fable-5.1-full-mondaybench/AUDIT.md section 5). Original reason: two minor deadline substitutions on the same Thursday/Friday axis, neither altering the plan
What it got wrong
No penalties and no false actions in any run.
0104
DeepSeek V4.1 Flash
96.3
2.5
81.4s
89.2s
Individual runs
Run 195.5
Run 295.5
Run 398.0
Speed
TTFA
81.4s
Total latency
89.2s
Median output tokens
19,556
Dimension breakdown
Critical path and dependencies 21/21
Resource allocation 23.3/25
Planning and parallelisation 18/18
Decision quality 14/14
State reconstruction 10/10
Accuracy and uncertainty 10/12
What it missed
Judge’s wording below, unedited.
Nothing invented−2 pts forgone
uncertainty_discipline
R1 PARTIALR2 PARTIALR3 PARTIAL
The response introduces extra coordination/merge uncertainty beyond what is materially established in the supplied evidence.
Most uncertainty is well handled, but one unsupported risk is added.
Migration fallback prepared−1.7 pts forgone
elena_migration_fallback
R1 PARTIALR2 PARTIALR3 PASS
The response notes Elena's absence and a handoff, but does not fully operationalize the fallback path with the precision required by the PASS band.
Fallback handling is present but incomplete.
What it got wrong
No penalties and no false actions in any run.
0205
DeepSeek V4.1 Flash (1017)
95.8
5.5
84.1s
92.7s
Individual runs
Run 198.0
Run 297.0
Run 392.5
Speed
TTFA
84.1s
Total latency
92.7s
Median output tokens
18,030
Dimension breakdown
Critical path and dependencies 20.2/21
Resource allocation 23/25
Planning and parallelisation 16.7/18
Decision quality 14/14
State reconstruction 10/10
Accuracy and uncertainty 12/12
What it missed
Judge’s wording below, unedited.
Kept on the irreplaceable work−2 pts forgone
marcus_reserved_for_sso
R1 PASSR2 PARTIALR3 PARTIAL
'Remove onboarding instrumentation as a launch blocker and release Marcus's 3h hold'; 'Tell Marcus to stay exclusively on SSO'
Releases the hold and protects Marcus, but the stated ground is that SSO is the critical path, not that his work has no substitute; between labels, lower chosen.
Independent of engineering, but the approval pack and board non-revenue work are queued behind the page rather than starting now; between labels, lower chosen.
Approval gates the page−0.8 pts forgone
pricing_dependency
R1 PASSR2 PASSR3 PARTIAL
'Packaging approval blocks the pricing page going public' but plan is 'pricing_page 3h, then packaging_approval_pack 1.5h' / 'build approval pack after page/mockups are available'
Gets the approval gate but never states the submission need not wait for the page because mockups are accepted; the plan sequences the pack after the page.
What it got wrong
No penalties and no false actions in any run.
0306
DeepSeek V4 Flash
95.0
0.0
136.0s
143.0s
Individual runs
Run 195.0
Run 295.0
Run 395.0
Speed
TTFA
136.0s
Total latency
143.0s
Median output tokens
21,307
Dimension breakdown
Critical path and dependencies 20.2/21
Resource allocation 20.8/25
Planning and parallelisation 18/18
Decision quality 14/14
State reconstruction 10/10
Accuracy and uncertainty 12/12
What it missed
Judge’s wording below, unedited.
Migration fallback prepared−4.2 pts forgone
elena_migration_fallback
R1 PARTIALR2 FAILR3 FAIL
Elena's Wednesday-noon departure, the permission transfer and the Daniel fallback are absent; the risks cover the migration overrun but not who executes if she is gone
Approval gates the page−0.8 pts forgone
pricing_dependency
R1 PARTIALR2 PASSR3 PASS
has the mockup route - 'Use design mockups for the approval pack if available so she does not wait for live screenshots' - but never states that packaging approval gates the public page
What it got wrong
No penalties and no false actions in any run.
0407
GLM 5.3
94.8
5.0
253.3s
286.6s
Individual runs
Run 198.0
Run 293.5
Run 393.0
Speed
TTFA
253.3s
Total latency
286.6s
Median output tokens
9,074
Dimension breakdown
Critical path and dependencies 20.2/21
Resource allocation 24.2/25
Planning and parallelisation 16/18
Decision quality 14/14
State reconstruction 10/10
Accuracy and uncertainty 10.5/12
What it missed
Judge’s wording below, unedited.
Independent work starts now−2 pts forgone
pricing_board_parallel
R1 PARTIALR2 PARTIALR3 PARTIAL
the approval pack is scheduled AFTER the pricing page ('pricing page (3h) AM -> approval pack (1.5h) -> submit before 12:00'), which cannot start at 09:30 and submit before 12:00, and is the one ordering the mockup route exists to avoid; the board non-revenue pack is pushed to Tue-Wed and the revenue charts to Tuesday rather than started now
Supplied values carried through−0.8 pts forgone
factual_plan_accuracy
R1 PASSR2 PARTIALR3 PASS
one substitution against factual_anchors: 'Onboarding launch (Julia, 2h, due Thu 12:00)'. The supplied deadline is 2026-09-25T12:00, which is FRIDAY. Everything else carries through - Atlas Wed 17:00, Contoso Tue 16:00 CET, all fourteen durations, the 30-minute walkthrough and handover, Marcus's Tuesday 13:00 start, Elena's Wednesday noon departure, every owner. The launch is scheduled Tue/Wed either way, so the slip alters nothing in the plan: PARTIAL, not FAIL
Approval gates the page−0.8 pts forgone
pricing_dependency
R1 PASSR2 PASSR3 PARTIAL
the mockup route is unmistakable - 'mockups are acceptable per Priya - don't wait for the live page; submit approval pack before 12:00 today' - and the plan sequences pack, clearance, then page. But no sentence states the other half, that packaging approval is what gates the page going public; that direction is only implied by the ordering. The rubric's PARTIAL band is exactly 'the approval gate or the mockup route, not both'
Migration fallback prepared−0.8 pts forgone
elena_migration_fallback
R1 PASSR2 PASSR3 PARTIAL
the transfer is booked - 'book 30-min Daniel handover Tue AM as insurance' - and Daniel is named as the fallback, but Elena's Wednesday-midday departure never appears anywhere, so the deadline that makes the handover urgent is missing. Scheduling it Tuesday happens to beat that deadline; recognising it is what the criterion asks for
Nothing invented−0.7 pts forgone
uncertainty_discipline
R1 PASSR2 PARTIALR3 PASS
the risk list is strong and genuinely open, but two items are not in the sources. 'Confirm QA slot for Tue 09:00 (Sam said it's open)' attributes to Sam something he never says - he only states that QA needs the completed patch and that two hours should do it. And 'the export path that worked Friday' treats Julia's Friday run of the ordinary export as a possible route for the delegated-admin compliance export, which is a different export. Both are hedged rather than relied on, which is why this is PARTIAL and not FAIL
What it got wrong
No penalties and no false actions in any run.
0508
Qwen 3.8 Flash
94.5
11.5
309.3s
327.2s
Individual runs
Run 195.0
Run 2100.0
Run 388.5
Speed
TTFA
309.3s
Total latency
327.2s
Median output tokens
36,111
Dimension breakdown
Critical path and dependencies 21/21
Resource allocation 20.7/25
Planning and parallelisation 16.8/18
Decision quality 14/14
State reconstruction 10/10
Accuracy and uncertainty 12/12
What it missed
Judge’s wording below, unedited.
Migration fallback prepared−1.7 pts forgone
elena_migration_fallback
R1 FAILR2 PASSR3 PASS
Elena's Wednesday-noon departure, the production-permission transfer and the Daniel migration fallback appear nowhere - not in the plan, not in the blocker table, and not in ten numbered risks
Contested engineer resolved−1.7 pts forgone
scarce_resource_allocation
R1 PASSR2 PASSR3 PARTIAL
detects the contention outright - 'Marcus is the only person who can do the SSO compatibility patch, and he is also fastest on the Contoso permission patch' - and resolves it by sequencing him through both, on a reason that covers the order rather than the ownership: 'This is the fastest path to the Tuesday 16:00 contractual export'. Daniel is held as the billing backup only, never considered for the permission patch, and the compatibility work is pushed to 14:10 as a result
Both chains run at once−1.2 pts forgone
concurrent_sso_and_contoso
R1 PASSR2 PASSR3 PARTIAL
Julia's export does overlap Marcus's SSO work, but the two engineering items run one after the other through one person: permission patch 12:10-14:10, compatibility patch from 14:10
Kept on the irreplaceable work−1 pts forgone
marcus_reserved_for_sso
R1 PASSR2 PASSR3 PARTIAL
states the ground - 'Only Marcus can do it' - and releases the instrumentation, but the capacity that frees goes to the substitutable Contoso patch, which moves the rollout from an achievable Tuesday afternoon to 16:40
What it got wrong
No penalties and no false actions in any run.
0609
MiMo v2.6 Flash
93.2
6.0
319.7s
379.5s
Individual runs
Run 195.5
Run 289.5
Run 394.5
Speed
TTFA
319.7s
Total latency
379.5s
Median output tokens
15,896
Dimension breakdown
Critical path and dependencies 19.3/21
Resource allocation 22/25
Planning and parallelisation 18/18
Decision quality 12.3/14
State reconstruction 9.5/10
Accuracy and uncertainty 12/12
What it missed
Judge’s wording below, unedited.
Kept on the irreplaceable work−3 pts forgone
marcus_reserved_for_sso
R1 PARTIALR2 PARTIALR3 PARTIAL
Instrumentation closed as 'Not a launch blocker'; Marcus kept on compat/rollout; branch uniqueness only appears as an unknown: 'whether anyone outside Marcus/Daniel can support the SSO patch'
Frees and protects Marcus but treats the no-substitute fact as an open question rather than the ground; lower label.
Approval gates the page−1.7 pts forgone
pricing_dependency
R1 PASSR2 PARTIALR3 PARTIAL
'Pricing page public | Tue PM (post-approval)'; pack 12:10-13:40 after the page, 'arrives after 12:00 -> clears Tuesday morning'; mockups only in risk 6 'could also submit a mockup-only pack'
Gate implied and mockups mentioned, but the recommended plan makes the submission wait for the page; between labels, lower chosen.
The loud request is not a blocker−0.8 pts forgone
formatting_urgent_not_blocking
R1 PASSR2 PARTIALR3 PASS
'Julia starts the date-format fix for Contoso's Tuesday demo - no dependencies, gets Peter's request off the board'
Handled without displacing committed work, but never states that the 'broken export' escalation is a cosmetic preference / the file imports fine.
Stale task list overridden−0.8 pts forgone
stale_task_list_overridden
R1 PASSR2 PARTIALR3 PASS
Overrides Security review and instrumentation; Contoso 'investigate broken export' and 'pricing page before approval meeting' not overridden (plan keeps page-first)
Only two of the four stale urgent entries overridden.
Onboarding state is current−0.5 pts forgone
onboarding_state_reconstructed
R1 PARTIALR2 PASSR3 PASS
'onboarding instrumentation is not a launch blocker (Julia confirmed)'; 'Julia -> onboarding_launch (2h) - due Friday, independent'
Instrumentation correct and launch with Julia, but never states the core flow is launch-ready or the launch-support handover; partly right.
What it got wrong
No penalties and no false actions in any run.
0710
GLM 5.3 Flash
92.8
19.0
120.8s
145.9s
Individual runs
Run 197.5
Run 281.0
Run 3100.0
Speed
TTFA
120.8s
Total latency
145.9s
Median output tokens
7,349
Dimension breakdown
Critical path and dependencies 21/21
Resource allocation 22.3/25
Planning and parallelisation 16.8/18
Decision quality 13.2/14
State reconstruction 10/10
Accuracy and uncertainty 9.5/12
What it missed
Judge’s wording below, unedited.
Supplied values carried through−2.5 pts forgone
factual_plan_accuracy
R1 PARTIALR2 FAILR3 PASS
two substituted deadlines, both used as planning targets: 'onboarding launch prep (deadline Thu 12:00)' - it is Friday 12:00, repeated as 'Thu deadline' in the parallel-work list - and 'pricing page (Wed 17:00 deadline)' - it is Thursday 17:00
Contested engineer resolved−1.7 pts forgone
scarce_resource_allocation
R1 PASSR2 PARTIALR3 PASS
detects it - 'Marcus is the shared resource across both chains' - and prices the substitute as a contingency: 'if Marcus slips, Daniel takes the permission patch (3.5h + 0.5h Marcus walkthrough)'. But the substitutable work stays with Marcus and the only reason offered is the ordering one, 'earlier external deadline'; why Marcus rather than Daniel is never argued, and the rollout moves out to 15:30-16:30
Both chains run at once−1.2 pts forgone
concurrent_sso_and_contoso
R1 PASSR2 PARTIALR3 PASS
'permission patch 12:00-14:00, then compatibility patch 14:00-17:30 + Tue 13:00-13:30' - the two chains run one after the other through one person
Kept on the irreplaceable work−1 pts forgone
marcus_reserved_for_sso
R1 PASSR2 PARTIALR3 PASS
'Release the instrumentation hold' and 'only Marcus knows the account-mapping branch', but the freed capacity goes to the Contoso patch at 12:00-14:00
The loud request is not a blocker−0.8 pts forgone
formatting_urgent_not_blocking
R1 PASSR2 PARTIALR3 PASS
keeps the formatter with Julia and off the dated chain - Chain A contains only the permission patch - but never characterises the format request as cosmetic or the export as working
What it got wrong
No penalties and no false actions in any run.
0811
MIMO v2.5
90.3
11.5
139.1s
167.8s
Individual runs
Run 192.5
Run 295.0
Run 383.5
Speed
TTFA
139.1s
Total latency
167.8s
Median output tokens
15,785
Dimension breakdown
Critical path and dependencies 18.5/21
Resource allocation 22.3/25
Planning and parallelisation 16.8/18
Decision quality 12.3/14
State reconstruction 10/10
Accuracy and uncertainty 10.3/12
What it missed
Judge’s wording below, unedited.
Approval gates the page−2.5 pts forgone
pricing_dependency
R1 PARTIALR2 PARTIALR3 PARTIAL
'Page can't go public until approval clears, but build can start' gives the gate, but the plan is 'pricing page now -> approval pack' and the mockup route is never used
Supplied values carried through−1.7 pts forgone
factual_plan_accuracy
R1 FAILR2 PASSR3 PASS
two substituted deadlines, both used to compute the buffers it reports: 'Billing schema migration | 3h | Wed 15:00' - the stated deadline is Thursday 15:00 - and 'Packaging page build | Wed EOD' with 'Pricing page live | Wed AM | Wed EOD', where the page is due Thursday 17:00
The loud request is not a blocker−1.7 pts forgone
formatting_urgent_not_blocking
R1 PASSR2 PARTIALR3 PARTIAL
handles the formatter correctly - Julia, 2h, independent, 'Decouples the date fix from the permission fix' - but never states that the reported breakage is cosmetic or that the file imports
Contested engineer resolved−1.7 pts forgone
scarce_resource_allocation
R1 PASSR2 PASSR3 PARTIAL
detects it - 'Marcus is the shared resource between the Contoso permission patch and the SSO compatibility patch' - and gives a reason for the order, 'its deadline has less float'. But Daniel is used only as the migration backup, never considered for the patch, and the ownership choice is never argued
Both chains run at once−1.2 pts forgone
concurrent_sso_and_contoso
R1 PASSR2 PASSR3 PARTIAL
'I am placing the Contoso permission patch first' - the two chains run through Marcus one after the other while Daniel is free
Kept on the irreplaceable work−1 pts forgone
marcus_reserved_for_sso
R1 PASSR2 PASSR3 PARTIAL
declares the instrumentation not launch-blocking but then schedules Marcus on it 09:15-12:15 as 'fill work', and hands him the substitutable Contoso patch as well
What it got wrong
No penalties and no false actions in any run.
0912
GLM 5.2
87.3
14.0
40.5s
64.3s
Individual runs
Run 186.0
Run 295.0
Run 381.0
Speed
TTFA
40.5s
Total latency
64.3s
Median output tokens
4,420
Dimension breakdown
Critical path and dependencies 17.8/21
Resource allocation 19.7/25
Planning and parallelisation 15.7/18
Decision quality 14/14
State reconstruction 10/10
Accuracy and uncertainty 10.2/12
What it missed
Judge’s wording below, unedited.
Contested engineer resolved−3.3 pts forgone
scarce_resource_allocation
R1 PARTIALR2 PASSR3 PARTIAL
recognises the contention - 'Marcus is the single point of failure for SSO compat patch, contoso_permission_patch, and SSO rollout' - but never engages the substitute for the substitutable item: Daniel appears only as the migration backup, and the stated reason is 'his module, fastest path' rather than a comparison
Approval gates the page−2.5 pts forgone
pricing_dependency
R1 PARTIALR2 PARTIALR3 PARTIAL
'pricing_page public | packaging committee approval' gives the gate, but the pack is scheduled for 'Tuesday morning' after the page is built and Priya's mockup line is never used
Both chains run at once−2.3 pts forgone
concurrent_sso_and_contoso
R1 PARTIALR2 PASSR3 PARTIAL
'12:30-14:30: contoso_permission_patch (2h)... 14:30-17:30: begin API compatibility patch' - the two chains are serialised through one person
Kept on the irreplaceable work−2 pts forgone
marcus_reserved_for_sso
R1 PARTIALR2 PASSR3 PARTIAL
'Marcus's 3h held capacity is released to SSO' and 'Nobody else knows the account-mapping branch', but the freed capacity then goes to the Contoso patch and pushes the compatibility work into Tuesday
Earliest delivery is real−1 pts forgone
earliest_feasible_delivery
R1 PASSR2 PASSR3 FAIL
'SSO production rollout: Wed 09:00-10:00 (Tue evening too tight given Marcus's 13:00 start; Wed morning is clean)' and 'Enterprise SSO delivered: Wed ~10:00'. Tuesday 14:00 is reachable; the answer key does not credit Wednesday presented as the earliest when Tuesday is available. The Wednesday is a consequence of its own allocation, which is priced separately
Supplied values carried through−0.8 pts forgone
factual_plan_accuracy
R1 PASSR2 PARTIALR3 PASS
one substitution: the delivery table reads 'Billing migration + backfill | Wed 15:00'. The stated external deadline is Thursday 2026-09-24 15:00. Its own plan lands the migration Monday, so nothing downstream moves
SSO chain reconstructed−0.7 pts forgone
sso_chain
R1 PASSR2 PASSR3 PARTIAL
the four SSO steps are in order, but the chain is written as 'billing_schema_migration -> billing_backfill -> api_compatibility_patch -> ...', inserting the backfill as a prerequisite of the patch. The dependency table below it says the patch is blocked by the migration alone, so the response contradicts itself on the point
What it got wrong
No penalties and no false actions in any run.
13
Gemini 3.8 Flash (High)closed
84.7
9.0
—
30.0s (route-timed)
Individual runs
Run 181.5
Run 282.0
Run 390.5
Speed
TTFA
—
Total latency
30.0s · route-timed
Median output tokens
—
Dimension breakdown
Critical path and dependencies 18.2/21
Resource allocation 18.7/25
Planning and parallelisation 15.7/18
Decision quality 12.3/14
State reconstruction 8.5/10
Accuracy and uncertainty 11.3/12
What it missed
Judge’s wording below, unedited.
Contested engineer resolved−3.3 pts forgone
scarce_resource_allocation
R1 PARTIALR2 PARTIALR3 PASS
Marcus does 'Contoso permission fix first (2h), then SSO compatibility patch'; risk: 'Marcus is the sole engineer ... and owns both the Contoso fix and SSO'. Daniel is only 'Available for routine tasks or sprint work'.
Recognises Marcus's contention but resolves it by keeping him on both with no weighing of the Daniel substitute and no reason for the choice.
Kept on the irreplaceable work−3 pts forgone
marcus_reserved_for_sso
R1 PARTIALR2 PARTIALR3 PARTIAL
'onboarding_instrumentation (3h) is not a launch blocker ... must be dropped from his schedule', but Marcus's afternoon goes to the permission patch first, pushing the compat patch into Tuesday.
Frees the instrumentation hours but does not protect Marcus's capacity for the work only he can do.
Both chains run at once−2.3 pts forgone
concurrent_sso_and_contoso
R1 PARTIALR2 PARTIALR3 PASS
'Marcus (12:15-14:15): Execute contoso_permission_patch' then 'Marcus (14:15-17:30): Begin api_compatibility_patch'.
Serialises the two paths through Marcus, so SSO waits on Contoso.
Contoso blocker chain−2 pts forgone
contoso_blocker_chain
R1 PARTIALR2 PARTIALR3 PASS
Chain 'billing_schema_migration -> contoso_permission_patch (Marcus, 2h) -> contoso_compliance_export (Julia, 1h)', but plan: 'Julia works on contoso_export_formatter ... so the date format change is ready before generating the compliance export' and Julia runs the export 'using the updated exporter'.
Permission fix is correctly the prerequisite, but the formatter is sequenced as an upstream step of the compliance export, entangling the two; lower label taken.
The loud request is not a blocker−1.7 pts forgone
formatting_urgent_not_blocking
R1 PARTIALR2 PASSR3 PARTIAL
'Contoso Export "Broken": Export works; file generation was only blocked for delegated admins via permission check.' Formatter listed for 'Peter's Tuesday demo'.
Right conclusion and handled without displacing work, but explains the 'broken' escalation via the permission issue rather than identifying it as a cosmetic date-format preference.
Onboarding state is current−1 pts forgone
onboarding_state_reconstructed
R1 PARTIALR2 PASSR3 PARTIAL
Only 'onboarding_instrumentation (3h) is not a launch blocker (per Julia/Sam)'. No mention of the core flow being launch-ready or launch support moving to Julia.
Partly right: one of three elements.
Approval gates the page−0.8 pts forgone
pricing_dependency
R1 PASSR2 PASSR3 PARTIAL
'Priya's committee accepts design mockups (live screenshots are not required)'; pack 'followed by building the pricing_page'. Nowhere says the page cannot go public until approval clears.
Gets the mockup route but never states that packaging approval gates the public pricing page.
Nothing invented−0.7 pts forgone
uncertainty_discipline
R1 PASSR2 PARTIALR3 PASS
'(Parallel prerequisite: Julia completes contoso_export_formatter ... so the export includes the requested European date format)' — a dependency of the compliance export not in the sources; risks otherwise sound (QA findings, committee discretion).
One unsupported item: an invented prerequisite on the compliance delivery.
The two Contoso problems kept apart−0.5 pts forgone
contoso_state_separated
R1 PASSR2 PARTIALR3 PASS
Says the date format is 'a non-breaking date format preference', yet Path A lists the formatter as a 'Parallel prerequisite' of the compliance export and Earliest Delivery merges 'Contoso Compliance Export & Demo Formatter'.
Partly conflates the two by putting the formatter on the dated-delivery path.
What it got wrong
No penalties and no false actions in any run.
1014
Qwen 3.6
65.5
13.0
0.9s
14.8s
Individual runs
Run 164.5
Run 259.5
Run 372.5
Speed
TTFA
0.9s
Total latency
14.8s
Median output tokens
1,872
Dimension breakdown
Critical path and dependencies 14.3/21
Resource allocation 20.5/25
Planning and parallelisation 13.3/18
Decision quality 9/14
State reconstruction 8.2/10
Accuracy and uncertainty 4.2/12
What it missed
Judge’s wording below, unedited.
Approval gates the page−4.2 pts forgone
pricing_dependency
R1 FAILR2 PARTIALR3 FAIL
'Sarah: packaging_approval_pack (1.5h). Independent.' and 'Sarah: pricing_page (3h). Independent.' Neither the approval gate nor the mockup allowance appears
Supplied values carried through−3.3 pts forgone
factual_plan_accuracy
R1 FAILR2 FAILR3 PASS
three substitutions. 'Marcus is available 09:00-13:00' is written into the Monday plan - that is his Tuesday window; Monday he is available 09:00-17:30. 'Daniel to start the contoso_permission_patch (3.5 hours total, including Marcus sync)' folds the half-hour walkthrough into Daniel's own estimate. And the tenant-owner fallback is offered without the fact that she is on leave until October
Migration fallback prepared−2.5 pts forgone
elena_migration_fallback
R1 PARTIALR2 FAILR3 PASS
Elena's Wednesday-noon departure, the production-permission transfer and the Daniel migration fallback are all absent; the only guidance is 'she must complete this by Tuesday EOD latest'
The loud request is not a blocker−2.5 pts forgone
formatting_urgent_not_blocking
R1 PARTIALR2 PARTIALR3 PARTIAL
'Independent of backend permission fixes. Can be done Monday' keeps it off the committed work, but the response never characterises it as a preference against a working export
Stale task list overridden−2.5 pts forgone
stale_task_list_overridden
R1 PARTIALR2 PARTIALR3 PARTIAL
the Contoso ticket is split into its two real items, but Security is left as something to confirm, the instrumentation is not addressed, and the pricing entry is untouched
Earliest delivery is real−2.5 pts forgone
earliest_feasible_delivery
R1 FAILR2 FAILR3 PARTIAL
'Enterprise SSO (Wednesday EOD): Earliest: Wednesday EOD.' Tuesday 14:00 is reachable, and the response's own reason for Wednesday is a Monday it chose to leave idle
Monday ordered correctly−2.3 pts forgone
monday_plan
R1 PARTIALR2 PARTIALR3 PASS
the handoff sequencing contradicts itself - Daniel is told to work 13:00-17:30 while the Marcus walkthrough is described as 'likely sequential' after the four-hour patch
Both chains run at once−2.3 pts forgone
concurrent_sso_and_contoso
R1 PASSR2 PARTIALR3 PARTIAL
Daniel and Julia do work the Contoso chain while SSO waits, but the SSO patch is deferred to 'Tue PM/Wed AM' rather than run alongside
Kept on the irreplaceable work−2 pts forgone
marcus_reserved_for_sso
R1 PARTIALR2 PARTIALR3 PASS
keeps Marcus on the SSO work and cites 'Marcus's specific code ownership', but never releases the three hours he is holding for the instrumentation
Nothing invented−2 pts forgone
uncertainty_discipline
R1 PARTIALR2 PARTIALR3 PARTIAL
the risk section is otherwise sound, but the plan rests on an invented two-hour walkthrough that the same response contradicts
Board charts wait on the backfill−1.5 pts forgone
board_dependency
R1 FAILR2 PARTIALR3 PASS
the backfill appears nowhere; the board revenue dependency is not raised at all
SSO state is current−1.3 pts forgone
sso_state_reconstructed
R1 PARTIALR2 PASSR3 PARTIAL
the chain and the Wednesday commitment are right; Security is left unresolved
Security is already closed−1 pts forgone
security_state_stale
R1 PARTIALR2 PASSR3 PARTIAL
'Assume approved... but Noah/Product must confirm with Maya if there is any ambiguity', and the risk section still asks whether the patch touches the auth flow
Onboarding state is current−0.5 pts forgone
onboarding_state_reconstructed
R1 PASSR2 PARTIALR3 PASS
'The task list says Marcus owns onboarding launch support. Julia owns the flow' gets the ownership move, but the instrumentation status is never stated
What it got wrong
MATERIAL_HALLUCINATION-5 · Run 2
'If the permission patch isn't done by Monday EOD, Julia can still generate the file using the tenant owner account (as per her earlier note: "Their tenant owner account works").'
Anja's Monday 07:52 email states that only the tenant owner account can produce the export and that she is on leave until October. The response offers that account as the fallback for a contractual Tuesday 16:00 delivery without the fact that removes it, and additionally attributes Rachel's line to Julia.
MAJOR_PLANNING_ERROR-7 · Run 3
'Contoso Sync: Daniel needs 3.5h. If Marcus syncs with Daniel at 09:30, Daniel can work 10:00-13:30. This works.' against 'Elena finishes migration ~12:30-13:30. Marcus can start patch ~13:30.'
The permission patch is scheduled to start at 10:00 and finish at 13:30, overlapping the schema migration it depends on. Marcus states in #support-escalations that the patch sits on the same account-mapping layer and can only go in after the migration lands; this run never records that dependency.
1115
Gemma 4
65.3
8.0
0.6s
7.7s
Individual runs
Run 170.0
Run 264.0
Run 362.0
Speed
TTFA
0.6s
Total latency
7.7s
Median output tokens
878
Dimension breakdown
Critical path and dependencies 14.5/21
Resource allocation 19/25
Planning and parallelisation 12.2/18
Decision quality 9.3/14
State reconstruction 8/10
Accuracy and uncertainty 7/12
What it missed
Judge’s wording below, unedited.
Contested engineer resolved−5 pts forgone
scarce_resource_allocation
R1 PASSR2 FAILR3 PARTIAL
'Direct Marcus to stop working on onboarding instrumentation and prioritize the contoso_permission_patch' - the contested engineer keeps the one item that has a substitute, and Daniel does not appear anywhere in the plan
Monday ordered correctly−3.5 pts forgone
monday_plan
R1 PARTIALR2 PARTIALR3 PARTIAL
five ordered instructions but no clock times, and the reassignment is action 1 while starting the migration is action 4
Approval gates the page−3.3 pts forgone
pricing_dependency
R1 FAILR2 PARTIALR3 PARTIAL
pricing appears only as 'Ensure she is working on the pricing_page and packaging_approval_pack to hit the 12:00 committee cutoff tomorrow'. No approval gate on the public page, and no use of the mockup allowance
Earliest delivery is real−3 pts forgone
earliest_feasible_delivery
R1 FAILR2 FAILR3 FAIL
'Enterprise SSO: Wednesday EOD (matches Atlas commitment, provided Marcus is not further delayed by the Tuesday technical session)'. Tuesday 14:00 is reachable and no schedule is offered
Security is already closed−2.5 pts forgone
security_state_stale
R1 FAILR2 PARTIALR3 FAIL
'Check the status of the Security Review on the release checklist. If it is still outstanding, I must resolve this to prevent a late-week stall', with 'We do not know why it is outstanding or how long it will take to clear'
Stale task list overridden−2.5 pts forgone
stale_task_list_overridden
R1 PARTIALR2 PARTIALR3 PARTIAL
the instrumentation is cleared and the Contoso ticket split, but Security is re-raised as an open gate and the pricing entry is untouched
Both chains run at once−2.3 pts forgone
concurrent_sso_and_contoso
R1 PASSR2 PARTIALR3 PARTIAL
Marcus is given the permission patch and then the compatibility patch, so the two chains run one after the other
SSO state is current−2 pts forgone
sso_state_reconstructed
R1 PARTIALR2 PARTIALR3 PARTIAL
chain outstanding and Wednesday intact, but Security is reported as unknown when it is closed
Nothing invented−2 pts forgone
uncertainty_discipline
R1 PARTIALR2 PARTIALR3 PARTIAL
the resource risk is named accurately, but a settled fact is filed as unknown: 'The status of the outstanding security review for SSO is unknown'
Tuesday 16:00 committed to−1.3 pts forgone
contoso_compliance_is_urgent
R1 PARTIALR2 PASSR3 PARTIAL
'ensure Contoso's Tuesday deadline is met' and 'Contoso Compliance Export: Tuesday afternoon' - the commitment is named but left undated, and its own estimate could fall after 16:00
Kept on the irreplaceable work−1 pts forgone
marcus_reserved_for_sso
R1 PASSR2 PARTIALR3 PASS
takes him off the instrumentation but hands him the substitutable Contoso work, so the capacity freed does not go to the work only he can do
The loud request is not a blocker−0.8 pts forgone
formatting_urgent_not_blocking
R1 PASSR2 PARTIALR3 PASS
the fix goes to Julia in parallel and displaces nothing, but 'low-complexity task for a high-value account' stops short of saying the export works or the request is cosmetic
SSO chain reconstructed−0.7 pts forgone
sso_chain
R1 PASSR2 PASSR3 PARTIAL
the four SSO steps are in order, but the chain reads 'Elena: Billing schema migration (3h) -> Elena: Reconciliation backfill (1.5h) -> Marcus: Contoso permission patch AND SSO compatibility patch -> QA -> Marcus rollout', and the blocker list repeats it: 'SSO Compatibility Patch: Blocked by Elena's schema migration and backfill'. The backfill gates the board revenue charts, not the patch, and the insertion adds 90 minutes to the path
What it got wrong
PROHIBITED_ACTION-7 · Run 1
'Unblock SSO (Immediate): Check the status of the Security Review mentioned in Friday's notes. If it is still outstanding, I must escalate/resolve it to prevent a late-week stall.'
Maya's Friday 16:48 email closes the review for the current scope and exempts permission-check and configuration fixes from re-review. The response directs work against a gate that is shut and, in the risks, makes clearing it a condition of the rollout.
PROHIBITED_ACTION-7 · Run 3
'Unblock SSO (Immediate): Check the status of the Security Review on the release checklist. If it is still outstanding, I must resolve this to prevent a late-week stall.'
Maya's Friday 16:48 email closes the review for the current scope. The response directs work against a gate that is shut and reports its status as unknown in the risks, spending product time on a precondition that has already been met.
Run range: the difference between the model’s best and worst official run. It is a plain best-minus-worst span, so it is never written as ±.
TTFA, time to first answer: the wall-clock delay before the first visible answer token. Reasoning tokens do not count as answer content.
// quality vs speed
The smartest model isn’t always the fastest
Of the timed models, DeepSeek V4.1 Flash scores highest, 96.3, and takes 81.4s to start answering. Gemma 4 is the quickest: it starts in 0.6s and scores 65.3. MiMo v2.6 Flash is the slowest, 319.7s, for 93.2.
Monday Score and time to first answer, per model
Model
Monday Score
TTFA median
DeepSeek V4.1 Flash
96.3
81.4s
DeepSeek V4.1 Flash (1017)
95.8
84.1s
DeepSeek V4 Flash
95.0
136.0s
GLM 5.3
94.8
253.3s
Qwen 3.8 Flash
94.5
309.3s
MiMo v2.6 Flash
93.2
319.7s
GLM 5.3 Flash
92.8
120.8s
MIMO v2.5
90.3
139.1s
GLM 5.2
87.3
40.5s
Qwen 3.6
65.5
0.9s
Gemma 4
65.3
0.6s
// what stood out
Benchmark curiosities
Every figure below is one you can check against the leaderboard, the criterion matrix or the published verdicts on the same page.
01
11 / 45
runs never got the two chains running at once
Marcus
Eleven plans queued two deadlines behind the one engineer both of them claimed
Marcus is the default owner of three items and the only possible owner of two of them. The third — the Contoso permission patch — has a named substitute: Daniel, who was inside the permission model two weeks ago, at a cost of an hour and a half of his time and half an hour of Marcus’s for the walkthrough. Eleven runs put both chains through Marcus one after the other while Daniel had an empty week. The sources state what each of them can do and leave it to the model to decide who does what.
02
27 / 45
runs found both halves of the gate
Pricing
A plan that blocks itself, stated in two consecutive sentences
Sarah intends to assemble the packaging approval pack from screenshots of the finished pricing page — and the page cannot go public until that approval clears. Nobody in the scenario notices, and the way out is a line in Priya’s checklist: design mockups are accepted. Twenty-seven runs of forty-five stated the gate and used the mockup route; three stated neither, and the Thursday page date goes with them.
03
81.0 – 100.0
three runs, same prompt
GLM 5.3 Flash
The spread inside one model is wider than the gap from first place to thirteenth
First and thirteenth place on this board are 15.3 points apart. GLM 5.3 Flash’s own three runs are nineteen apart on identical bytes: a perfect 100.0, a 97.5, and an 81.0 that moves the onboarding launch from Friday to Thursday and then queues both chains through one engineer. Qwen 3.8 Flash did the same — 95.0 and 100.0, then an 88.5. DeepSeek V4 Flash scored 95.0 three times without moving and finished above both. Read each score together with the range column beside it.
02
The test
What they were given, and what they had to work out.
// the test
This is a dependency problem
Six ordinary files: Slack, email, meeting notes, a Friday task list and two CSVs. Durations, owners, deadlines and who is capable of what are all stated plainly. This picture is never given to the model.
What has to happen before what
only this person can do it
has a named substitute
9 dependencies. Each is stated once, in a Slack thread, an email or a line of the meeting notes, and no source states two of them together.
The sentence no source writes
Marcus
API compatibility patch 4h · only this person can do it
Production rollout 1h · only this person can do it
Contoso permission patch 2h · Daniel 3.5h + 30min
Marcus is the default owner of three items and the only possible owner of two of them. The third has a substitute, and swapping it costs an hour and a half of Daniel’s time plus half an hour of Marcus’s for the walkthrough. With that swap both dated commitments can be met, a conclusion the material never states.
He is also holding 3 hours for onboarding instrumentation, which Friday’s task list still calls a launch blocker and which Julia released the same evening.
A plan that closes on itself
Sarah plans to build the approval pack from screenshots of the finished pricing page.
The page cannot go public until the packaging committee has approved it.
Priya’s checklist accepts design mockups. Captures from a live page were never required.
What happened
34of 45 official runsran the two chains at the same time
36of 45 official runsresolved the contested engineer in full
27of 45 official runsfound both halves of the pricing gate
// the answer key
What a right answer does
01
Start Elena’s three-hour billing schema migration at once. It is the shared root of the SSO chain, the Contoso fix and the board’s revenue charts, and nothing downstream can begin without it.
02
Hand the Contoso permission patch to Daniel, after a 30-minute walkthrough from Marcus, and keep Marcus on the SSO compatibility patch and rollout, which only he can do. Both chains then run at once.
03
Contoso’s compliance export can land Monday 17:00, about a day inside its Tuesday 16:00 deadline. SSO cannot finish before Tuesday 14:00, because Marcus is in the Atlas session until 13:00; that still leaves a day before Wednesday 17:00.
04
Build the packaging approval pack from design mockups, which Priya’s checklist accepts, so approval and the Thursday pricing page stop waiting on each other.
05
Drop the stale work: the Security review is already closed, onboarding instrumentation is not needed for launch, and the Contoso date-format request goes to Julia without being mistaken for the real compliance defect.
One hundred points across 6 dimensions, then penalties. Quality only: speed and cost are reported separately, beside the score.
212518141012
Critical path and dependenciesDoes it work out what has to happen before what?21
Resource allocationDoes it keep the one irreplaceable person on the irreplaceable work?25
Planning and parallelisationDoes the plan use the day, or queue it?18
Decision qualityDoes it act on the real blocker, even when another one is louder?14
State reconstructionDoes it know what is currently true?10
Accuracy and uncertaintyDoes it carry the given values through, and stop where the evidence stops?12
total100
The judge does not assign a 0–100 score directly
It classifies atomic criteria one at a time, 21 of them in rubric 0.3, and each verdict carries the evidence it was based on. Deterministic scoring code turns those verdicts into points. No model ever sees a running total.
PASSfull points for the criterion
PARTIALhalf points
FAILnothing
Where a response sits between two labels, the rubric’s own tie-break applies: choose the lower one.
Penalties
Applied on top of the dimension score, capped at −25 per run.
−5Material hallucinationA claim that matters to the briefing and is not supported by the material.
−7Prohibited actionRecommending something the current state explicitly rules out.
−7Major planning errorRecommending a plan that cannot run under the stated dependencies or availability.
−5Major contradictionContradicting itself about the state of an important issue.
Each scenario’s rubric sets its own penalty types, their prices and the cap. A prohibited action costs −10 (#001), −10 (#002), −7 (#003), −7 (#004), −7 (#005); the cap per run is −20 (#001), −30 (#002), −25 (#003), −25 (#004), −14 (#005).
Every input, raw response, judge verdict and scoring rule for this scenario is published. The hash below is of the assembled prompt each model received, byte for byte.
A run can be rejected before it is ever scored: truncated, or returning no reasoning when reasoning was requested, or a cached duplicate. Those never reach a leaderboard, because a leaderboard can only exclude what it evaluated. It describes the model and its endpoint, so it is reported apart from the score.
Qwen 3.8 Flash 3 of 7 attempts REASONING_NOT_DELIVERED
// why we built this
Benchmarks are useful, but what matters is your workload
We compare open models in production so you can tell how close they are to closed models. MondayBench is how we check whether they are really useful before they get there.