New: DeepSeek V4.1 Flash.The architecture built for agents, and the 890 bytes that change an agent bill.Explore the piece →
// mondaybench #003beta
We gave 10 open models the same messy Monday morning
Six sources. Six dated commitments across one week, three of them waiting on the same three-hour migration, and one engineer claimed by two of them at once.
// six dated commitments · one week · 3 waits on the billing migration
now · Monday 09:10
waits on the billing migration
owes it nothing
01
The result
Ten models, 30 runs, one judge.
// results
The leaderboard
Monday Score is quality only: how well the briefing reconstructed the situation and picked the work that mattered. Speed sits beside it and is never folded in.
Open models onlyAll models
monday-001 · 13 models, 3 runs each
#
Model
Monday Score
Run range
TTFA median
Total latency
Show detail for
1
GPT-6 Astragpt-6-astraclosed
100.0
0.0
—
—
Individual runs
Run 1100.0
Run 2100.0
Run 3100.0
Speed
TTFA
—
Total latency
—
Median output tokens
—
Dimension breakdown
21/21
25/25
18/18
14/14
State reconstruction 10/10
12/12
What it missed
Nothing. Full marks on every criterion.
What it got wrong
No penalties and no false actions in any run.
2
Claude Opus 5closed
98.3
5.0
–
–
3
Fable 5.1closed
96.7
5.0
–
–
14
DeepSeek V4.1 Flashdeepseek-v4.1-flash-rerun-v1
96.3
2.5
81.4s
89.2s
Individual runs
Run 195.5
Run 295.5
Run 398.0
Speed
TTFA
81.4s
Total latency
89.2s
Median output tokens
19,556
Dimension breakdown
21/21
23.3/25
18/18
14/14
State reconstruction 10/10
10/12
What it missed
uncertainty_discipline−2 pts forgone
Nothing invented
PARTIALPARTIALPARTIAL
The response introduces extra coordination/merge uncertainty beyond what is materially established in the supplied evidence.
Most uncertainty is well handled, but one unsupported risk is added.
elena_migration_fallback−1.7 pts forgone
Migration fallback prepared
PARTIALPARTIALPASS
The response notes Elena's absence and a handoff, but does not fully operationalize the fallback path with the precision required by the PASS band.
Fallback handling is present but incomplete.
What it got wrong
No penalties and no false actions in any run.
25
DeepSeek V4 Flashdeepseek-v4-flash
95.0
0.0
136.0s
143.0s
Individual runs
Run 195.0
Run 295.0
Run 395.0
Speed
TTFA
136.0s
Total latency
143.0s
Median output tokens
21,307
Dimension breakdown
20.2/21
20.8/25
18/18
14/14
State reconstruction 10/10
12/12
What it missed
elena_migration_fallback−4.2 pts forgone
Migration fallback prepared
PARTIALFAILFAIL
Elena's Wednesday-noon departure, the permission transfer and the Daniel fallback are absent; the risks cover the migration overrun but not who executes if she is gone
Elena's Wednesday-noon departure, the permission transfer and the Daniel fallback are absent; the risks cover the migration overrun but not who executes if she is gone
pricing_dependency−0.8 pts forgone
Approval gates the page
PARTIALPASSPASS
has the mockup route - 'Use design mockups for the approval pack if available so she does not wait for live screenshots' - but never states that packaging approval gates the public page
has the mockup route - 'Use design mockups for the approval pack if available so she does not wait for live screenshots' - but never states that packaging approval gates the public page
What it got wrong
No penalties and no false actions in any run.
36
GLM 5.3glm5.3
94.8
5.0
253.3s
286.6s
Individual runs
Run 198.0
Run 293.5
Run 393.0
Speed
TTFA
253.3s
Total latency
286.6s
Median output tokens
9,074
Dimension breakdown
20.2/21
24.2/25
16/18
14/14
State reconstruction 10/10
10.5/12
What it missed
pricing_board_parallel−2 pts forgone
Independent work starts now
PARTIALPARTIALPARTIAL
the approval pack is scheduled AFTER the pricing page ('pricing page (3h) AM -> approval pack (1.5h) -> submit before 12:00'), which cannot start at 09:30 and submit before 12:00, and is the one ordering the mockup route exists to avoid; the board non-revenue pack is pushed to Tue-Wed and the revenue charts to Tuesday rather than started now
the approval pack is scheduled AFTER the pricing page ('pricing page (3h) AM -> approval pack (1.5h) -> submit before 12:00'), which cannot start at 09:30 and submit before 12:00, and is the one ordering the mockup route exists to avoid; the board non-revenue pack is pushed to Tue-Wed and the revenue charts to Tuesday rather than started now
factual_plan_accuracy−0.8 pts forgone
Supplied values carried through
PASSPARTIALPASS
one substitution against factual_anchors: 'Onboarding launch (Julia, 2h, due Thu 12:00)'. The supplied deadline is 2026-09-25T12:00, which is FRIDAY. Everything else carries through - Atlas Wed 17:00, Contoso Tue 16:00 CET, all fourteen durations, the 30-minute walkthrough and handover, Marcus's Tuesday 13:00 start, Elena's Wednesday noon departure, every owner. The launch is scheduled Tue/Wed either way, so the slip alters nothing in the plan: PARTIAL, not FAIL
one substitution against factual_anchors: 'Onboarding launch (Julia, 2h, due Thu 12:00)'. The supplied deadline is 2026-09-25T12:00, which is FRIDAY. Everything else carries through - Atlas Wed 17:00, Contoso Tue 16:00 CET, all fourteen durations, the 30-minute walkthrough and handover, Marcus's Tuesday 13:00 start, Elena's Wednesday noon departure, every owner. The launch is scheduled Tue/Wed either way, so the slip alters nothing in the plan: PARTIAL, not FAIL
pricing_dependency−0.8 pts forgone
Approval gates the page
PASSPASSPARTIAL
the mockup route is unmistakable - 'mockups are acceptable per Priya - don't wait for the live page; submit approval pack before 12:00 today' - and the plan sequences pack, clearance, then page. But no sentence states the other half, that packaging approval is what gates the page going public; that direction is only implied by the ordering. The rubric's PARTIAL band is exactly 'the approval gate or the mockup route, not both'
the mockup route is unmistakable - 'mockups are acceptable per Priya - don't wait for the live page; submit approval pack before 12:00 today' - and the plan sequences pack, clearance, then page. But no sentence states the other half, that packaging approval is what gates the page going public; that direction is only implied by the ordering. The rubric's PARTIAL band is exactly 'the approval gate or the mockup route, not both'
elena_migration_fallback−0.8 pts forgone
Migration fallback prepared
PASSPASSPARTIAL
the transfer is booked - 'book 30-min Daniel handover Tue AM as insurance' - and Daniel is named as the fallback, but Elena's Wednesday-midday departure never appears anywhere, so the deadline that makes the handover urgent is missing. Scheduling it Tuesday happens to beat that deadline; recognising it is what the criterion asks for
the transfer is booked - 'book 30-min Daniel handover Tue AM as insurance' - and Daniel is named as the fallback, but Elena's Wednesday-midday departure never appears anywhere, so the deadline that makes the handover urgent is missing. Scheduling it Tuesday happens to beat that deadline; recognising it is what the criterion asks for
uncertainty_discipline−0.7 pts forgone
Nothing invented
PASSPARTIALPASS
the risk list is strong and genuinely open, but two items are not in the sources. 'Confirm QA slot for Tue 09:00 (Sam said it's open)' attributes to Sam something he never says - he only states that QA needs the completed patch and that two hours should do it. And 'the export path that worked Friday' treats Julia's Friday run of the ordinary export as a possible route for the delegated-admin compliance export, which is a different export. Both are hedged rather than relied on, which is why this is PARTIAL and not FAIL
the risk list is strong and genuinely open, but two items are not in the sources. 'Confirm QA slot for Tue 09:00 (Sam said it's open)' attributes to Sam something he never says - he only states that QA needs the completed patch and that two hours should do it. And 'the export path that worked Friday' treats Julia's Friday run of the ordinary export as a possible route for the delegated-admin compliance export, which is a different export. Both are hedged rather than relied on, which is why this is PARTIAL and not FAIL
What it got wrong
No penalties and no false actions in any run.
47
Qwen 3.8 Flashqwen3.8-flash
94.5
11.5
309.3s
327.2s
Individual runs
Run 195.0
Run 2100.0
Run 388.5
Speed
TTFA
309.3s
Total latency
327.2s
Median output tokens
36,111
Dimension breakdown
21/21
20.7/25
16.8/18
14/14
State reconstruction 10/10
12/12
What it missed
elena_migration_fallback−1.7 pts forgone
Migration fallback prepared
FAILPASSPASS
Elena's Wednesday-noon departure, the production-permission transfer and the Daniel migration fallback appear nowhere - not in the plan, not in the blocker table, and not in ten numbered risks
Elena's Wednesday-noon departure, the production-permission transfer and the Daniel migration fallback appear nowhere - not in the plan, not in the blocker table, and not in ten numbered risks
scarce_resource_allocation−1.7 pts forgone
Contested engineer resolved
PASSPASSPARTIAL
detects the contention outright - 'Marcus is the only person who can do the SSO compatibility patch, and he is also fastest on the Contoso permission patch' - and resolves it by sequencing him through both, on a reason that covers the order rather than the ownership: 'This is the fastest path to the Tuesday 16:00 contractual export'. Daniel is held as the billing backup only, never considered for the permission patch, and the compatibility work is pushed to 14:10 as a result
detects the contention outright - 'Marcus is the only person who can do the SSO compatibility patch, and he is also fastest on the Contoso permission patch' - and resolves it by sequencing him through both, on a reason that covers the order rather than the ownership: 'This is the fastest path to the Tuesday 16:00 contractual export'. Daniel is held as the billing backup only, never considered for the permission patch, and the compatibility work is pushed to 14:10 as a result
concurrent_sso_and_contoso−1.2 pts forgone
Both chains run at once
PASSPASSPARTIAL
Julia's export does overlap Marcus's SSO work, but the two engineering items run one after the other through one person: permission patch 12:10-14:10, compatibility patch from 14:10
Julia's export does overlap Marcus's SSO work, but the two engineering items run one after the other through one person: permission patch 12:10-14:10, compatibility patch from 14:10
marcus_reserved_for_sso−1 pts forgone
Kept on the irreplaceable work
PASSPASSPARTIAL
states the ground - 'Only Marcus can do it' - and releases the instrumentation, but the capacity that frees goes to the substitutable Contoso patch, which moves the rollout from an achievable Tuesday afternoon to 16:40
states the ground - 'Only Marcus can do it' - and releases the instrumentation, but the capacity that frees goes to the substitutable Contoso patch, which moves the rollout from an achievable Tuesday afternoon to 16:40
What it got wrong
No penalties and no false actions in any run.
58
GLM 5.3 Flashglm5.3-flash
92.8
19.0
120.8s
145.9s
Individual runs
Run 197.5
Run 281.0
Run 3100.0
Speed
TTFA
120.8s
Total latency
145.9s
Median output tokens
7,349
Dimension breakdown
21/21
22.3/25
16.8/18
13.2/14
State reconstruction 10/10
9.5/12
What it missed
factual_plan_accuracy−2.5 pts forgone
Supplied values carried through
PARTIALFAILPASS
two substituted deadlines, both used as planning targets: 'onboarding launch prep (deadline Thu 12:00)' - it is Friday 12:00, repeated as 'Thu deadline' in the parallel-work list - and 'pricing page (Wed 17:00 deadline)' - it is Thursday 17:00
two substituted deadlines, both used as planning targets: 'onboarding launch prep (deadline Thu 12:00)' - it is Friday 12:00, repeated as 'Thu deadline' in the parallel-work list - and 'pricing page (Wed 17:00 deadline)' - it is Thursday 17:00
scarce_resource_allocation−1.7 pts forgone
Contested engineer resolved
PASSPARTIALPASS
detects it - 'Marcus is the shared resource across both chains' - and prices the substitute as a contingency: 'if Marcus slips, Daniel takes the permission patch (3.5h + 0.5h Marcus walkthrough)'. But the substitutable work stays with Marcus and the only reason offered is the ordering one, 'earlier external deadline'; why Marcus rather than Daniel is never argued, and the rollout moves out to 15:30-16:30
detects it - 'Marcus is the shared resource across both chains' - and prices the substitute as a contingency: 'if Marcus slips, Daniel takes the permission patch (3.5h + 0.5h Marcus walkthrough)'. But the substitutable work stays with Marcus and the only reason offered is the ordering one, 'earlier external deadline'; why Marcus rather than Daniel is never argued, and the rollout moves out to 15:30-16:30
concurrent_sso_and_contoso−1.2 pts forgone
Both chains run at once
PASSPARTIALPASS
'permission patch 12:00-14:00, then compatibility patch 14:00-17:30 + Tue 13:00-13:30' - the two chains run one after the other through one person
'permission patch 12:00-14:00, then compatibility patch 14:00-17:30 + Tue 13:00-13:30' - the two chains run one after the other through one person
marcus_reserved_for_sso−1 pts forgone
Kept on the irreplaceable work
PASSPARTIALPASS
'Release the instrumentation hold' and 'only Marcus knows the account-mapping branch', but the freed capacity goes to the Contoso patch at 12:00-14:00
'Release the instrumentation hold' and 'only Marcus knows the account-mapping branch', but the freed capacity goes to the Contoso patch at 12:00-14:00
formatting_urgent_not_blocking−0.8 pts forgone
The loud request is not a blocker
PASSPARTIALPASS
keeps the formatter with Julia and off the dated chain - Chain A contains only the permission patch - but never characterises the format request as cosmetic or the export as working
keeps the formatter with Julia and off the dated chain - Chain A contains only the permission patch - but never characterises the format request as cosmetic or the export as working
What it got wrong
No penalties and no false actions in any run.
69
MIMO v2.5mimo-v2.5
90.3
11.5
139.1s
167.8s
Individual runs
Run 192.5
Run 295.0
Run 383.5
Speed
TTFA
139.1s
Total latency
167.8s
Median output tokens
15,785
Dimension breakdown
18.5/21
22.3/25
16.8/18
12.3/14
State reconstruction 10/10
10.3/12
What it missed
pricing_dependency−2.5 pts forgone
Approval gates the page
PARTIALPARTIALPARTIAL
'Page can't go public until approval clears, but build can start' gives the gate, but the plan is 'pricing page now -> approval pack' and the mockup route is never used
'Page can't go public until approval clears, but build can start' gives the gate, but the plan is 'pricing page now -> approval pack' and the mockup route is never used
factual_plan_accuracy−1.7 pts forgone
Supplied values carried through
FAILPASSPASS
two substituted deadlines, both used to compute the buffers it reports: 'Billing schema migration | 3h | Wed 15:00' - the stated deadline is Thursday 15:00 - and 'Packaging page build | Wed EOD' with 'Pricing page live | Wed AM | Wed EOD', where the page is due Thursday 17:00
two substituted deadlines, both used to compute the buffers it reports: 'Billing schema migration | 3h | Wed 15:00' - the stated deadline is Thursday 15:00 - and 'Packaging page build | Wed EOD' with 'Pricing page live | Wed AM | Wed EOD', where the page is due Thursday 17:00
formatting_urgent_not_blocking−1.7 pts forgone
The loud request is not a blocker
PASSPARTIALPARTIAL
handles the formatter correctly - Julia, 2h, independent, 'Decouples the date fix from the permission fix' - but never states that the reported breakage is cosmetic or that the file imports
handles the formatter correctly - Julia, 2h, independent, 'Decouples the date fix from the permission fix' - but never states that the reported breakage is cosmetic or that the file imports
scarce_resource_allocation−1.7 pts forgone
Contested engineer resolved
PASSPASSPARTIAL
detects it - 'Marcus is the shared resource between the Contoso permission patch and the SSO compatibility patch' - and gives a reason for the order, 'its deadline has less float'. But Daniel is used only as the migration backup, never considered for the patch, and the ownership choice is never argued
detects it - 'Marcus is the shared resource between the Contoso permission patch and the SSO compatibility patch' - and gives a reason for the order, 'its deadline has less float'. But Daniel is used only as the migration backup, never considered for the patch, and the ownership choice is never argued
concurrent_sso_and_contoso−1.2 pts forgone
Both chains run at once
PASSPASSPARTIAL
'I am placing the Contoso permission patch first' - the two chains run through Marcus one after the other while Daniel is free
'I am placing the Contoso permission patch first' - the two chains run through Marcus one after the other while Daniel is free
marcus_reserved_for_sso−1 pts forgone
Kept on the irreplaceable work
PASSPASSPARTIAL
declares the instrumentation not launch-blocking but then schedules Marcus on it 09:15-12:15 as 'fill work', and hands him the substitutable Contoso patch as well
declares the instrumentation not launch-blocking but then schedules Marcus on it 09:15-12:15 as 'fill work', and hands him the substitutable Contoso patch as well
What it got wrong
No penalties and no false actions in any run.
710
GLM 5.2glm5.2
87.3
14.0
40.5s
64.3s
Individual runs
Run 186.0
Run 295.0
Run 381.0
Speed
TTFA
40.5s
Total latency
64.3s
Median output tokens
4,420
Dimension breakdown
17.8/21
19.7/25
15.7/18
14/14
State reconstruction 10/10
10.2/12
What it missed
scarce_resource_allocation−3.3 pts forgone
Contested engineer resolved
PARTIALPASSPARTIAL
recognises the contention - 'Marcus is the single point of failure for SSO compat patch, contoso_permission_patch, and SSO rollout' - but never engages the substitute for the substitutable item: Daniel appears only as the migration backup, and the stated reason is 'his module, fastest path' rather than a comparison
recognises the contention - 'Marcus is the single point of failure for SSO compat patch, contoso_permission_patch, and SSO rollout' - but never engages the substitute for the substitutable item: Daniel appears only as the migration backup, and the stated reason is 'his module, fastest path' rather than a comparison
pricing_dependency−2.5 pts forgone
Approval gates the page
PARTIALPARTIALPARTIAL
'pricing_page public | packaging committee approval' gives the gate, but the pack is scheduled for 'Tuesday morning' after the page is built and Priya's mockup line is never used
'pricing_page public | packaging committee approval' gives the gate, but the pack is scheduled for 'Tuesday morning' after the page is built and Priya's mockup line is never used
concurrent_sso_and_contoso−2.3 pts forgone
Both chains run at once
PARTIALPASSPARTIAL
'12:30-14:30: contoso_permission_patch (2h)... 14:30-17:30: begin API compatibility patch' - the two chains are serialised through one person
'12:30-14:30: contoso_permission_patch (2h)... 14:30-17:30: begin API compatibility patch' - the two chains are serialised through one person
marcus_reserved_for_sso−2 pts forgone
Kept on the irreplaceable work
PARTIALPASSPARTIAL
'Marcus's 3h held capacity is released to SSO' and 'Nobody else knows the account-mapping branch', but the freed capacity then goes to the Contoso patch and pushes the compatibility work into Tuesday
'Marcus's 3h held capacity is released to SSO' and 'Nobody else knows the account-mapping branch', but the freed capacity then goes to the Contoso patch and pushes the compatibility work into Tuesday
earliest_feasible_delivery−1 pts forgone
Earliest delivery is real
PASSPASSFAIL
'SSO production rollout: Wed 09:00-10:00 (Tue evening too tight given Marcus's 13:00 start; Wed morning is clean)' and 'Enterprise SSO delivered: Wed ~10:00'. Tuesday 14:00 is reachable; the answer key does not credit Wednesday presented as the earliest when Tuesday is available. The Wednesday is a consequence of its own allocation, which is priced separately
'SSO production rollout: Wed 09:00-10:00 (Tue evening too tight given Marcus's 13:00 start; Wed morning is clean)' and 'Enterprise SSO delivered: Wed ~10:00'. Tuesday 14:00 is reachable; the answer key does not credit Wednesday presented as the earliest when Tuesday is available. The Wednesday is a consequence of its own allocation, which is priced separately
factual_plan_accuracy−0.8 pts forgone
Supplied values carried through
PASSPARTIALPASS
one substitution: the delivery table reads 'Billing migration + backfill | Wed 15:00'. The stated external deadline is Thursday 2026-09-24 15:00. Its own plan lands the migration Monday, so nothing downstream moves
one substitution: the delivery table reads 'Billing migration + backfill | Wed 15:00'. The stated external deadline is Thursday 2026-09-24 15:00. Its own plan lands the migration Monday, so nothing downstream moves
sso_chain−0.7 pts forgone
SSO chain reconstructed
PASSPASSPARTIAL
the four SSO steps are in order, but the chain is written as 'billing_schema_migration -> billing_backfill -> api_compatibility_patch -> ...', inserting the backfill as a prerequisite of the patch. The dependency table below it says the patch is blocked by the migration alone, so the response contradicts itself on the point
the four SSO steps are in order, but the chain is written as 'billing_schema_migration -> billing_backfill -> api_compatibility_patch -> ...', inserting the backfill as a prerequisite of the patch. The dependency table below it says the patch is blocked by the migration alone, so the response contradicts itself on the point
What it got wrong
No penalties and no false actions in any run.
11
Gemini 3.8 Flash (High)closed
85.3
23.5
–
30.0s
812
Qwen 3.6qwen3.6
65.5
13.0
0.9s
14.8s
Individual runs
Run 164.5
Run 259.5
Run 372.5
Speed
TTFA
0.9s
Total latency
14.8s
Median output tokens
1,872
Dimension breakdown
14.3/21
20.5/25
13.3/18
9/14
State reconstruction 8.2/10
4.2/12
What it missed
pricing_dependency−4.2 pts forgone
Approval gates the page
FAILPARTIALFAIL
'Sarah: packaging_approval_pack (1.5h). Independent.' and 'Sarah: pricing_page (3h). Independent.' Neither the approval gate nor the mockup allowance appears
'Sarah: packaging_approval_pack (1.5h). Independent.' and 'Sarah: pricing_page (3h). Independent.' Neither the approval gate nor the mockup allowance appears
factual_plan_accuracy−3.3 pts forgone
Supplied values carried through
FAILFAILPASS
three substitutions. 'Marcus is available 09:00-13:00' is written into the Monday plan - that is his Tuesday window; Monday he is available 09:00-17:30. 'Daniel to start the contoso_permission_patch (3.5 hours total, including Marcus sync)' folds the half-hour walkthrough into Daniel's own estimate. And the tenant-owner fallback is offered without the fact that she is on leave until October
three substitutions. 'Marcus is available 09:00-13:00' is written into the Monday plan - that is his Tuesday window; Monday he is available 09:00-17:30. 'Daniel to start the contoso_permission_patch (3.5 hours total, including Marcus sync)' folds the half-hour walkthrough into Daniel's own estimate. And the tenant-owner fallback is offered without the fact that she is on leave until October
elena_migration_fallback−2.5 pts forgone
Migration fallback prepared
PARTIALFAILPASS
Elena's Wednesday-noon departure, the production-permission transfer and the Daniel migration fallback are all absent; the only guidance is 'she must complete this by Tuesday EOD latest'
Elena's Wednesday-noon departure, the production-permission transfer and the Daniel migration fallback are all absent; the only guidance is 'she must complete this by Tuesday EOD latest'
formatting_urgent_not_blocking−2.5 pts forgone
The loud request is not a blocker
PARTIALPARTIALPARTIAL
'Independent of backend permission fixes. Can be done Monday' keeps it off the committed work, but the response never characterises it as a preference against a working export
'Independent of backend permission fixes. Can be done Monday' keeps it off the committed work, but the response never characterises it as a preference against a working export
stale_task_list_overridden−2.5 pts forgone
Stale task list overridden
PARTIALPARTIALPARTIAL
the Contoso ticket is split into its two real items, but Security is left as something to confirm, the instrumentation is not addressed, and the pricing entry is untouched
the Contoso ticket is split into its two real items, but Security is left as something to confirm, the instrumentation is not addressed, and the pricing entry is untouched
earliest_feasible_delivery−2.5 pts forgone
Earliest delivery is real
FAILFAILPARTIAL
'Enterprise SSO (Wednesday EOD): Earliest: Wednesday EOD.' Tuesday 14:00 is reachable, and the response's own reason for Wednesday is a Monday it chose to leave idle
'Enterprise SSO (Wednesday EOD): Earliest: Wednesday EOD.' Tuesday 14:00 is reachable, and the response's own reason for Wednesday is a Monday it chose to leave idle
monday_plan−2.3 pts forgone
Monday ordered correctly
PARTIALPARTIALPASS
the handoff sequencing contradicts itself - Daniel is told to work 13:00-17:30 while the Marcus walkthrough is described as 'likely sequential' after the four-hour patch
the handoff sequencing contradicts itself - Daniel is told to work 13:00-17:30 while the Marcus walkthrough is described as 'likely sequential' after the four-hour patch
concurrent_sso_and_contoso−2.3 pts forgone
Both chains run at once
PASSPARTIALPARTIAL
Daniel and Julia do work the Contoso chain while SSO waits, but the SSO patch is deferred to 'Tue PM/Wed AM' rather than run alongside
Daniel and Julia do work the Contoso chain while SSO waits, but the SSO patch is deferred to 'Tue PM/Wed AM' rather than run alongside
marcus_reserved_for_sso−2 pts forgone
Kept on the irreplaceable work
PARTIALPARTIALPASS
keeps Marcus on the SSO work and cites 'Marcus's specific code ownership', but never releases the three hours he is holding for the instrumentation
keeps Marcus on the SSO work and cites 'Marcus's specific code ownership', but never releases the three hours he is holding for the instrumentation
uncertainty_discipline−2 pts forgone
Nothing invented
PARTIALPARTIALPARTIAL
the risk section is otherwise sound, but the plan rests on an invented two-hour walkthrough that the same response contradicts
the risk section is otherwise sound, but the plan rests on an invented two-hour walkthrough that the same response contradicts
board_dependency−1.5 pts forgone
Board charts wait on the backfill
FAILPARTIALPASS
the backfill appears nowhere; the board revenue dependency is not raised at all
the backfill appears nowhere; the board revenue dependency is not raised at all
sso_state_reconstructed−1.3 pts forgone
SSO state is current
PARTIALPASSPARTIAL
the chain and the Wednesday commitment are right; Security is left unresolved
the chain and the Wednesday commitment are right; Security is left unresolved
security_state_stale−1 pts forgone
Security is already closed
PARTIALPASSPARTIAL
'Assume approved... but Noah/Product must confirm with Maya if there is any ambiguity', and the risk section still asks whether the patch touches the auth flow
'Assume approved... but Noah/Product must confirm with Maya if there is any ambiguity', and the risk section still asks whether the patch touches the auth flow
onboarding_state_reconstructed−0.5 pts forgone
Onboarding state is current
PASSPARTIALPASS
'The task list says Marcus owns onboarding launch support. Julia owns the flow' gets the ownership move, but the instrumentation status is never stated
'The task list says Marcus owns onboarding launch support. Julia owns the flow' gets the ownership move, but the instrumentation status is never stated
What it got wrong
MATERIAL_HALLUCINATION-5 · Run 2
'If the permission patch isn't done by Monday EOD, Julia can still generate the file using the tenant owner account (as per her earlier note: "Their tenant owner account works").'
Anja's Monday 07:52 email states that only the tenant owner account can produce the export and that she is on leave until October. The response offers that account as the fallback for a contractual Tuesday 16:00 delivery without the fact that removes it, and additionally attributes Rachel's line to Julia.
MAJOR_PLANNING_ERROR-7 · Run 3
'Contoso Sync: Daniel needs 3.5h. If Marcus syncs with Daniel at 09:30, Daniel can work 10:00-13:30. This works.' against 'Elena finishes migration ~12:30-13:30. Marcus can start patch ~13:30.'
The permission patch is scheduled to start at 10:00 and finish at 13:30, overlapping the schema migration it depends on. Marcus states in #support-escalations that the patch sits on the same account-mapping layer and can only go in after the migration lands; this run never records that dependency.
913
Gemma 4gemma4
65.3
8.0
0.6s
7.7s
Individual runs
Run 170.0
Run 264.0
Run 362.0
Speed
TTFA
0.6s
Total latency
7.7s
Median output tokens
878
Dimension breakdown
14.5/21
19/25
12.2/18
9.3/14
State reconstruction 8/10
7/12
What it missed
scarce_resource_allocation−5 pts forgone
Contested engineer resolved
PASSFAILPARTIAL
'Direct Marcus to stop working on onboarding instrumentation and prioritize the contoso_permission_patch' - the contested engineer keeps the one item that has a substitute, and Daniel does not appear anywhere in the plan
'Direct Marcus to stop working on onboarding instrumentation and prioritize the contoso_permission_patch' - the contested engineer keeps the one item that has a substitute, and Daniel does not appear anywhere in the plan
monday_plan−3.5 pts forgone
Monday ordered correctly
PARTIALPARTIALPARTIAL
five ordered instructions but no clock times, and the reassignment is action 1 while starting the migration is action 4
five ordered instructions but no clock times, and the reassignment is action 1 while starting the migration is action 4
pricing_dependency−3.3 pts forgone
Approval gates the page
FAILPARTIALPARTIAL
pricing appears only as 'Ensure she is working on the pricing_page and packaging_approval_pack to hit the 12:00 committee cutoff tomorrow'. No approval gate on the public page, and no use of the mockup allowance
pricing appears only as 'Ensure she is working on the pricing_page and packaging_approval_pack to hit the 12:00 committee cutoff tomorrow'. No approval gate on the public page, and no use of the mockup allowance
earliest_feasible_delivery−3 pts forgone
Earliest delivery is real
FAILFAILFAIL
'Enterprise SSO: Wednesday EOD (matches Atlas commitment, provided Marcus is not further delayed by the Tuesday technical session)'. Tuesday 14:00 is reachable and no schedule is offered
'Enterprise SSO: Wednesday EOD (matches Atlas commitment, provided Marcus is not further delayed by the Tuesday technical session)'. Tuesday 14:00 is reachable and no schedule is offered
security_state_stale−2.5 pts forgone
Security is already closed
FAILPARTIALFAIL
'Check the status of the Security Review on the release checklist. If it is still outstanding, I must resolve this to prevent a late-week stall', with 'We do not know why it is outstanding or how long it will take to clear'
'Check the status of the Security Review on the release checklist. If it is still outstanding, I must resolve this to prevent a late-week stall', with 'We do not know why it is outstanding or how long it will take to clear'
stale_task_list_overridden−2.5 pts forgone
Stale task list overridden
PARTIALPARTIALPARTIAL
the instrumentation is cleared and the Contoso ticket split, but Security is re-raised as an open gate and the pricing entry is untouched
the instrumentation is cleared and the Contoso ticket split, but Security is re-raised as an open gate and the pricing entry is untouched
concurrent_sso_and_contoso−2.3 pts forgone
Both chains run at once
PASSPARTIALPARTIAL
Marcus is given the permission patch and then the compatibility patch, so the two chains run one after the other
Marcus is given the permission patch and then the compatibility patch, so the two chains run one after the other
sso_state_reconstructed−2 pts forgone
SSO state is current
PARTIALPARTIALPARTIAL
chain outstanding and Wednesday intact, but Security is reported as unknown when it is closed
chain outstanding and Wednesday intact, but Security is reported as unknown when it is closed
uncertainty_discipline−2 pts forgone
Nothing invented
PARTIALPARTIALPARTIAL
the resource risk is named accurately, but a settled fact is filed as unknown: 'The status of the outstanding security review for SSO is unknown'
the resource risk is named accurately, but a settled fact is filed as unknown: 'The status of the outstanding security review for SSO is unknown'
contoso_compliance_is_urgent−1.3 pts forgone
Tuesday 16:00 committed to
PARTIALPASSPARTIAL
'ensure Contoso's Tuesday deadline is met' and 'Contoso Compliance Export: Tuesday afternoon' - the commitment is named but left undated, and its own estimate could fall after 16:00
'ensure Contoso's Tuesday deadline is met' and 'Contoso Compliance Export: Tuesday afternoon' - the commitment is named but left undated, and its own estimate could fall after 16:00
marcus_reserved_for_sso−1 pts forgone
Kept on the irreplaceable work
PASSPARTIALPASS
takes him off the instrumentation but hands him the substitutable Contoso work, so the capacity freed does not go to the work only he can do
takes him off the instrumentation but hands him the substitutable Contoso work, so the capacity freed does not go to the work only he can do
formatting_urgent_not_blocking−0.8 pts forgone
The loud request is not a blocker
PASSPARTIALPASS
the fix goes to Julia in parallel and displaces nothing, but 'low-complexity task for a high-value account' stops short of saying the export works or the request is cosmetic
the fix goes to Julia in parallel and displaces nothing, but 'low-complexity task for a high-value account' stops short of saying the export works or the request is cosmetic
sso_chain−0.7 pts forgone
SSO chain reconstructed
PASSPASSPARTIAL
the four SSO steps are in order, but the chain reads 'Elena: Billing schema migration (3h) -> Elena: Reconciliation backfill (1.5h) -> Marcus: Contoso permission patch AND SSO compatibility patch -> QA -> Marcus rollout', and the blocker list repeats it: 'SSO Compatibility Patch: Blocked by Elena's schema migration and backfill'. The backfill gates the board revenue charts, not the patch, and the insertion adds 90 minutes to the path
the four SSO steps are in order, but the chain reads 'Elena: Billing schema migration (3h) -> Elena: Reconciliation backfill (1.5h) -> Marcus: Contoso permission patch AND SSO compatibility patch -> QA -> Marcus rollout', and the blocker list repeats it: 'SSO Compatibility Patch: Blocked by Elena's schema migration and backfill'. The backfill gates the board revenue charts, not the patch, and the insertion adds 90 minutes to the path
What it got wrong
PROHIBITED_ACTION-7 · Run 1
'Unblock SSO (Immediate): Check the status of the Security Review mentioned in Friday's notes. If it is still outstanding, I must escalate/resolve it to prevent a late-week stall.'
Maya's Friday 16:48 email closes the review for the current scope and exempts permission-check and configuration fixes from re-review. The response directs work against a gate that is shut and, in the risks, makes clearing it a condition of the rollout.
PROHIBITED_ACTION-7 · Run 3
'Unblock SSO (Immediate): Check the status of the Security Review on the release checklist. If it is still outstanding, I must resolve this to prevent a late-week stall.'
Maya's Friday 16:48 email closes the review for the current scope. The response directs work against a gate that is shut and reports its status as unknown in the risks, spending product time on a precondition that has already been met.
Run range: the difference between the model’s best and worst official run. Not a deviation from the mean, so it is never written as ±.
TTFA, time to first answer: the wall-clock delay before the first visible answer token. Reasoning tokens do not count as answer content.
// quality vs speed
The smartest model isn’t always the one you want to wait for
GPT-6 Astra tops this scenario at 100.0, after — before its first answer token. GPT-6 Astra starts answering in — and scores 100.0. Models trade quality for response time in very different ways.
Two independent axes. There is no combined score, and there will not be one: how good the briefing is and how long you wait for it are different questions, and which matters more depends on what you are building.
Monday Score and time to first answer, per model
Model
Monday Score
TTFA median
GPT-6 Astra
100.0
—
DeepSeek V4.1 Flash
96.3
81.4s
DeepSeek V4 Flash
95.0
136.0s
GLM 5.3
94.8
253.3s
Qwen 3.8 Flash
94.5
309.3s
GLM 5.3 Flash
92.8
120.8s
MIMO v2.5
90.3
139.1s
GLM 5.2
87.3
40.5s
Qwen 3.6
65.5
0.9s
Gemma 4
65.3
0.6s
// what stood out
Three things worth saying out loud
Every figure below is one you can check against the leaderboard, the criterion matrix or the published verdicts on the same page.
01
9 / 24
runs never got the two chains running at once
Marcus
One engineer was claimed by two deadlines. Nine plans queued them behind him
Marcus is the default owner of three items and the only possible owner of two of them. The third — the Contoso permission patch — has a named substitute: Daniel, who was inside the permission model two weeks ago, at a cost of an hour and a half of his time and half an hour of Marcus’s for the walkthrough. Nine runs put both chains through Marcus one after the other while Daniel had an empty week. The sources state what each of them can do; not one of them states who should do what.
02
10 / 24
runs found both halves of the gate
Pricing
A plan that blocks itself, stated in two consecutive sentences
Sarah intends to assemble the packaging approval pack from screenshots of the finished pricing page — and the page cannot go public until that approval clears. Nobody in the scenario notices, and the way out is a line in Priya’s checklist: design mockups are accepted. Ten runs of twenty-four stated the gate and used the mockup route; three stated neither, and the Thursday page date goes with them.
03
81.0 – 100.0
three runs, same prompt
GLM 5.3 Flash
The spread inside one model is wider than the gap across the top five
Fewer than five points separate first place from fifth. GLM 5.3 Flash’s own three runs are nineteen apart on identical bytes: a perfect 100.0, a 97.5, and an 81.0 that moves the onboarding launch from Friday to Thursday and then queues both chains through one engineer. Qwen 3.8 Flash lost first place the same way — 100.0 and 95.0, then an 88.5. DeepSeek V4 Flash won it by scoring 95.0 three times without moving. Read the range column beside the score, or do not read the score.
02
The test
What they were given, and what they had to work out.
// the test
Every fact is written down Not one conclusion is
Six ordinary files: Slack, email, meeting notes, a Friday task list and two CSVs. Durations, owners, deadlines and who is capable of what are all stated plainly. What the model is never given is this picture.
What has to happen before what
only this person can do it
has a named substitute
9 dependencies. Each is stated once, in a Slack thread, an email or a line of the meeting notes, and no source states two of them together.
The sentence no source writes
Marcus
API compatibility patch 4h · only this person can do it
Production rollout 1h · only this person can do it
Contoso permission patch 2h · Daniel 3.5h + 30min
Marcus is the default owner of three items and the only possible owner of two of them. The third has a substitute, and swapping it costs an hour and a half of Daniel’s time plus half an hour of Marcus’s for the walkthrough. Both dated commitments can then be met. Nothing in the material says so.
He is also holding 3 hours for onboarding instrumentation, which Friday’s task list still calls a launch blocker and which Julia released the same evening.
A plan that closes on itself
Sarah plans to build the approval pack from screenshots of the finished pricing page.
The page cannot go public until the packaging committee has approved it.
Priya’s checklist accepts design mockups. Captures from a live page were never required.
What happened
15of 24 official runsran the two chains at the same time
17of 24 official runsresolved the contested engineer in full
10of 24 official runsfound both halves of the pricing gate
03
The method
How the score is built, and how to check it.
// scoring
How Monday Score works
One hundred points across 6 dimensions, then penalties. Quality only: speed and cost are reported beside the score and never inside it.
212518141012
Critical path and dependenciesDoes it work out what has to happen before what?21
Resource allocationDoes it keep the one irreplaceable person on the irreplaceable work?25
Planning and parallelisationDoes the plan use the day, or queue it?18
Decision qualityDoes it act on the real blocker rather than the loudest one?14
State reconstructionDoes it know what is currently true?10
Accuracy and uncertaintyDoes it carry the given values through, and stop where the evidence stops?12
total100
The judge does not assign a 0–100 score directly
It classifies atomic criteria one at a time, 21 of them in rubric 0.3, and each verdict carries the evidence it was based on. Deterministic scoring code turns those verdicts into points. No model ever sees a running total.
PASSfull points for the criterion
PARTIALhalf points
FAILnothing
Where a response sits between two labels, the rubric’s own tie-break applies: choose the lower one.
Penalties
Applied on top of the dimension score, capped at −25 per run.
-5Material hallucinationA claim that matters to the briefing and is not supported by the material.
-7Prohibited actionRecommending something the current state explicitly rules out.
-7Major planning errorRecommending a plan that cannot run under the stated dependencies or availability.
-5Major contradictionContradicting itself about the state of an important issue.
Every input, raw response, judge verdict and scoring rule for this scenario is published. The hash below is of the assembled prompt each model received, byte for byte.
A run can be rejected before it is ever scored: truncated, or returning no reasoning when reasoning was requested, or a cached duplicate. Those never reach a leaderboard, because a leaderboard can only exclude what it evaluated. This is a property of the model and its endpoint rather than of the answer, and it is reported apart from the score.
Qwen 3.8 Flash 3 of 7 attempts REASONING_NOT_DELIVERED
// why we built this
Benchmarks are useful Production is the point
We run open models in production for European teams. MondayBench is how we test whether they are actually useful before they get there.