// mondaybench #004 beta

El mismo lunes caótico
para 15 modelos

Ocho ficheros, siete de ellos tablas. Dos meses de ingresos, uso, churn, captación y soporte de una empresa SaaS, y tres decisiones que esperan a números que no están en ninguna tabla.

Una pregunta: ¿Razona sobre datos de negocio sucios?

ver_resultados ver_metodologia

15 modelos
3 runs cada uno
juez Opus 5
congelado y auditable

siete tablas

customers.csv 36 filas customer_id segmento, plan, ARR, renovación, estado
revenue.csv 71 filas customer_id facturas, dos meses, dos fuentes
usage.csv 67 filas usage_account_id peticiones, tokens, usuarios activos
plans.csv 6 filas plan ingresos, coste de infra y soporte, cuota incluida
acquisition.csv 15 filas customer_id canal, logo nuevo o expansión, ARR atribuido
funnel.csv 8 filas channel leads y pruebas de pago, por canal y mes
support.csv 133 filas customer_id tickets, severidad, estado

usage_account_id cruza por otra clave: dos cuentas la escriben de otra forma

El resultado

Quince modelos, 45 runs, un juez.

// resultados

La clasificación

El Monday Score mide solo calidad: cómo respondió cada modelo a «¿Razona sobre datos de negocio sucios?», criterio a criterio contra la rúbrica de este escenario. La velocidad va al lado y nunca se mezcla dentro.

Todos los modelos Solo modelos de pesos abiertos

monday-004 · 15 modelos, 3 runs cada uno
#	Modelo	Monday Score	Rango de runs	TTFA mediana	Latencia total
01 01	GLM 5.3 Flash	98.0	1.5	766.2s	827.8s
Runs individuales Run 1 98.5 Run 2 98.5 Run 3 97.0 Velocidad TTFA 766.2s Latencia total 827.8s Mediana de tokens de salida 36,248 Desglose por dimensión Razonamiento de ingresos 20/20 Salud de clientes 15/15 Razonamiento de captación 19/19 Economía de los planes 19/19 Calidad del dato entre tablas 16.5/18 Calidad de la decisión 8.5/9 Qué se dejó Palabras del juez, sin editar y en su inglés original. Arrastra sus propias cifras sin corromperlas −1.5 pts perdidos `figures_carried_correctly` R1 PARTIALR2 PARTIALR3 PARTIAL 'wonka ... 7 tickets in October - 5 of them rate_limits' when 4 of the 7 are rate_limits; and 'wayne renews in ~5 weeks' when 2026-11-30 is four weeks after the review date Two small slips in supporting detail. Everything else checkable is exact, including the EUR 262.9k -> EUR 246.5k profit walk, the +5.8% cost growth, the MRR bridge, the EUR 216k expansion bookings itemised, the EUR 3.23M of ARR with open high-severity tickets, all eight funnel cells, the full new-logo table, all twelve plan figures, contribution per customer for all three plans, and the mid-market -EUR 42k walk. No recomienda recortar el segmento −0.5 pts perdidos `segment_recommendation` R1 PASSR2 PASSR3 PARTIAL 'Yes - reduce acquisition investment in SMB, redirect rather than cut ... pause incremental paid SMB acquisition, redirect budget to SMB retention/onboarding ... Revisit once CAC-by-segment exists.' A reduction hedged with conditions, which is the PARTIAL band precisely. It names the missing input the PASS bar asks for and the budget stays inside SMB rather than leaving it, but leadership is still told to pause acquisition spend. En qué se equivocó Sin penalizaciones ni falsas acciones en ningún run.
02	GPT-6 Astra cerrado	98.0	0.0	—	—
Runs individuales Run 1 98.0 Run 2 98.0 Run 3 98.0 Velocidad TTFA — Latencia total — Mediana de tokens de salida — Desglose por dimensión Razonamiento de ingresos 20/20 Salud de clientes 15/15 Razonamiento de captación 19/19 Economía de los planes 19/19 Calidad del dato entre tablas 16/18 Calidad de la decisión 9/9 Qué se dejó Palabras del juez, sin editar y en su inglés original. Compara la carga de soporte como tasa normalizada −2 pts perdidos `support_normalized` R1 PARTIALR2 PARTIALR3 PARTIAL PARTIAL: Pro's ticket movement is quantified (19 -> 28, rate limits 5 -> 13) and support cost sits per plan in the contribution table, but no rate is computed on any denominator and Enterprise's raw share of tickets is never addressed En qué se equivocó Sin penalizaciones ni falsas acciones en ningún run.
02 03	Qwen 3.8 Flash	96.5	3.0	394.1s	468.0s
Runs individuales Run 1 96.5 Run 2 98.0 Run 3 95.0 Velocidad TTFA 394.1s Latencia total 468.0s Mediana de tokens de salida 52,295 Desglose por dimensión Razonamiento de ingresos 20/20 Salud de clientes 15/15 Razonamiento de captación 19/19 Economía de los planes 19/19 Calidad del dato entre tablas 15/18 Calidad de la decisión 8.5/9 Qué se dejó Palabras del juez, sin editar y en su inglés original. Compara la carga de soporte como tasa normalizada −2 pts perdidos `support_normalized` R1 PARTIALR2 PARTIALR3 PARTIAL 'October support tickets by plan: Enterprise 34 -> 35, Pro 19 -> 28, Starter 7 -> 11' and 'September Enterprise-segment tickets: 26, October: 25' It avoids the trap - Enterprise is called stable and Pro gets the focus - and it reads the segment view as well as the plan view, which is more care than most. But no rate against any denominator is computed, so the normalisation the criterion asks for is not performed. Arrastra sus propias cifras sin corromperlas −1 pts perdidos `figures_carried_correctly` R1 PARTIALR2 PASSR3 PARTIAL 'Enterprise 34 September tickets' where the enterprise plan has 33, which also makes its own column sum to 60 against a September total of 59 One miscount. Everything else checkable is exact, and the response carries more arithmetic than any other in the battery: the ARR bridge reconciling to EUR 8,788,800 to the euro, four revenue bases, the segment ARR walk including the reconstructed EUR 1,704,000, both channel tables with totals, the full two-month plan economics with cost subtotals, the utilisation table computed from customer counts, all five over-allowance accounts with overage volumes, the Soylent four-metric table, and the 13 October rate-limit tickets itemised per account - the only response anywhere to get that count right. No recomienda recortar el segmento −0.5 pts perdidos `segment_recommendation` R1 PASSR2 PASSR3 PARTIAL 'the data do not support a full cut without CAC and segment cost data' and 'Reallocate SMB investment toward onboarding, usage activation, and retention measurement rather than cutting the segment entirely' - but also 'For SMB, reduce or pause low-quality acquisition spend, especially starter acquisition, until activation and retention improve.' A reduction hedged with conditions, which is the PARTIAL band. It names the missing inputs the PASS bar asks for and keeps the budget inside SMB, but leadership is still told to pause acquisition spend rather than to investigate first. En qué se equivocó Sin penalizaciones ni falsas acciones en ningún run.
03 04	GLM 5.3	91.7	17.0	634.2s	673.2s
Runs individuales Run 1 95.0 Run 2 81.5 Run 3 98.5 Velocidad TTFA 634.2s Latencia total 673.2s Mediana de tokens de salida 46,592 Desglose por dimensión Razonamiento de ingresos 19/20 Salud de clientes 14.2/15 Razonamiento de captación 19/19 Economía de los planes 17.3/19 Calidad del dato entre tablas 13.7/18 Calidad de la decisión 8.5/9 Qué se dejó Palabras del juez, sin editar y en su inglés original. Arrastra sus propias cifras sin corromperlas −2 pts perdidos `figures_carried_correctly` R1 PARTIALR2 FAILR3 PARTIAL five incorrect figures across different parts of the analysis, one of them a headline that contradicts its own components. The ARR bridge reads 'EUR8,953k to EUR8,789k = -EUR164k (-1.8%)' while the components listed beneath it (+246, +133, +36, -30, -304) sum to +EUR81k; the closing figure is right, the opening figure is unreachable, and the stated direction is wrong - ARR on 1 September was about EUR8,713k, so it rose. Also: Acme is put at '84%' of its 3M included requests when 2.31M is 77%; '4 of 12 September SMB logos' where there are 13; rate-limit tickets 'tripled (4 to 13)' where September is 5; and 'infra per token ~EUR11.7/M' which divides the enterprise PLAN's infra cost by the enterprise SEGMENT's tokens, the plan-consistent figure being about EUR9.7/M - a wrong secondary rate. Repeated incorrect figures across different parts, plus one materially wrong: the FAIL band Compara la carga de soporte como tasa normalizada −1.3 pts perdidos `support_normalized` R1 PARTIALR2 PARTIALR3 PASS support cost as a share of plan revenue is computed for STARTER only - '32% of revenue (EUR792/customer/month vs EUR2,462 ARPU)', both exact. Enterprise's support is discussed only as a margin movement and Pro's only as a month-on-month change; the raw ticket counts by plan never appear, so the conclusion that Enterprise is not the worst despite raising 35 of 74 October tickets is never reached. That is the rubric's 'computes a rate for one plan only' Describe el patrón sin afirmar una causa −1 pts perdidos `causal_boundary` R1 PARTIALR2 PARTIALR3 PASS all three Soylent movements are present and correct - '-27%' usage, 'tickets went 3 to 8', and the EUR2.5k downgrade - and no mechanism is asserted or implied. What the PASS band also requires is the explicit statement that the data does not establish what caused what, and the response never makes it for Soylent; the epistemic caveat it does give is about channel attribution Da el titular y una cifra normalizada −1 pts perdidos `headline_vs_recurring_growth` R1 PASSR2 PARTIALR3 PASS the normalised figure is right and its treatment is stated - EUR746.1k to EUR757.7k, +1.6% - and the gap is correctly attributed to the annual invoice and the duplicate. But the headline it contrasts against is the DEDUPLICATED +9.6%; the raw invoice sum's ~+12.0% is never computed. EUR892.9k appears once, in the data-issues section, as a total rather than as a growth rate. That is the rubric's 'gives only one of the two figures' Cuantifica la retención sobre una base declarada −0.8 pts perdidos `segment_retention_quantified` R1 PASSR2 PARTIALR3 PASS SMB is correctly identified as worst and Enterprise as zero, but the middle is never quantified: no mid-market churn rate appears anywhere, so the three populations cannot be ranked from the figures given. The two framings are also mixed - '44% of the starter base' is plan, '4 of 12 September SMB logos' is segment - and that segment denominator is wrong (13 SMB accounts predate October, not 12). Neither error changes which population is worst, which is the PARTIAL band Separa la contribución absoluta del margen −0.8 pts perdidos `absolute_vs_scalable_economics` R1 PASSR2 PARTIALR3 PASS the contrast is drawn correctly - '87% of revenue, 79% of gross profit - but margin fell 2.7pp' against 'Pro is the margin leader (62.7%)' - and then the product recommendation leans on the absolute figure anyway: 'Plan focus (Product): Enterprise plan ... It carries 79% of gross profit'. That is the PARTIAL band's second clause word for word Respalda el foco de producto con dos señales además del margen −0.8 pts perdidos `product_focus_supported` R1 PASSR2 PARTIALR3 PASS the Enterprise plan is chosen over Pro. The other-plan route is open in the rubric, but only when argued on cost-adjusted economics and at least two independent signals AND not resting on absolute revenue. Margin erosion and the open high-severity tickets are real signals; the case is nevertheless led by 'it carries 79% of gross profit' and closed by 'one enterprise churn (stark = EUR1.14M) would erase more ARR than a full quarter of new logos', both absolute-size arguments. Pro is named only as a secondary track No recomienda recortar el segmento −0.5 pts perdidos `segment_recommendation` R1 PASSR2 PARTIALR3 PASS SMB is not cut and the missing input is named ('Segment P&L cannot be built: costs are attributed by plan, and plans cut across segments'), but the recommendation is a reduction with conditions attached: 'freeze incremental SMB acquisition spend ... set a January checkpoint - cut only if SMB net revenue retention hasn't improved' En qué se equivocó Sin penalizaciones ni falsas acciones en ningún run.
05	Claude Opus 5 cerrado	90.8	9.5	—	—
Runs individuales Run 1 86.5 Run 2 90.0 Run 3 96.0 Velocidad TTFA — Latencia total — Mediana de tokens de salida 23,925 Desglose por dimensión Razonamiento de ingresos 20/20 Salud de clientes 12.5/15 Razonamiento de captación 18.2/19 Economía de los planes 16.3/19 Calidad del dato entre tablas 15.2/18 Calidad de la decisión 8.7/9 Qué se dejó Palabras del juez, sin editar y en su inglés original. Cuantifica la retención sobre una base declarada −2.5 pts perdidos `segment_retention_quantified` R1 PARTIALR2 PARTIALR3 PARTIAL Quantifies some churn populations (notably Starter/SMB and the Nakatomi loss) but does not provide a complete, consistently-labelled retention/churn rate across enough segment/plan populations to support the rubric's full ranking bar. Separa la contribución absoluta del margen −1.7 pts perdidos `absolute_vs_scalable_economics` R1 PARTIALR2 PARTIALR3 PASS Presents Enterprise's absolute contribution and Pro's stronger margin, but does not clearly distinguish the decision question answered by absolute scale versus scalable economics/revenue per customer. Arrastra sus propias cifras sin corromperlas −1.5 pts perdidos `figures_carried_correctly` R1 PARTIALR2 PARTIALR3 PARTIAL Contains minor secondary quantitative/base slips that do not change the recommendations; the main revenue, margin, acquisition and plan conclusions still stand on correct figures. Compara la carga de soporte como tasa normalizada −1.3 pts perdidos `support_normalized` R1 PARTIALR2 PARTIALR3 PASS Uses ticket shares/allocations inside Enterprise and Pro rate-limit counts, but does not make a clean normalized support-load comparison across plans. Lee a los clientes que hay detrás de la media del plan −1 pts perdidos `plan_trajectory_evidence` R1 PARTIALR2 PARTIALR3 PASS Gives strong quantified Pro allowance/rate-limit evidence, but lacks a consistently quantified forward-looking usage signal for each plan. Rehace el ranking de canales solo con logos nuevos −0.8 pts perdidos `existing_customer_expansion_removed` R1 PARTIALR2 PASSR3 PASS Correctly strips paid-search expansion and identifies Partner as strong, but does not fully recompute/state the Partner €166.8k > Content €91.2k > Paid Search €64.8k new-logo ordering. Nombra las lagunas reales y no inventa cifras −0.3 pts perdidos `uncertainty_discipline` R1 PARTIALR2 PASSR3 PASS Names the major gaps, but also introduces unsupported/hypothetical quantitative extrapolations (for example a hypothetical Starter CAC tolerance/payback range) alongside known minor figure errors; lower-label rule => PARTIAL. En qué se equivocó Sin penalizaciones ni falsas acciones en ningún run.
04 06	DeepSeek V4.1 Flash	90.0	1.0	108.3s	128.6s
Runs individuales Run 1 89.5 Run 2 90.0 Run 3 90.5 Velocidad TTFA 108.3s Latencia total 128.6s Mediana de tokens de salida 30,057 Desglose por dimensión Razonamiento de ingresos 20/20 Salud de clientes 15/15 Razonamiento de captación 19/19 Economía de los planes 12.5/19 Calidad del dato entre tablas 18/18 Calidad de la decisión 5.5/9 Qué se dejó Palabras del juez, sin editar y en su inglés original. Separa la contribución absoluta del margen −2.5 pts perdidos `absolute_vs_scalable_economics` R1 PARTIALR2 PARTIALR3 PARTIAL The response correctly distinguishes Enterprise absolute profit from Pro margin, but then leans on Enterprise's absolute pool when choosing product focus. The contrast is calculated but not kept binding in the final decision. Respalda el foco de producto con dos señales además del margen −2.5 pts perdidos `product_focus_supported` R1 PARTIALR2 PARTIALR3 PARTIAL The response chooses Enterprise first and Pro second; the choice leans materially on Enterprise's absolute profit pool. It does not cleanly select the scalable-economics signal the rubric is designed to test. No recomienda recortar el segmento −2 pts perdidos `segment_recommendation` R1 FAILR2 PARTIALR3 PARTIAL The response says 'The data supports reducing new acquisition investment in SMB/Starter.' The pack lacks segment CAC/LTV/retention economics sufficient to recommend cutting or defunding SMB. Lee a los clientes que hay detrás de la media del plan −1.5 pts perdidos `plan_trajectory_evidence` R1 PARTIALR2 PARTIALR3 PARTIAL Usage/adoption trajectory is discussed, but the forward-looking plan comparison is not fully quantified across the plans. Trajectory evidence is useful but incomplete. Nombra las lagunas reales y no inventa cifras −1 pts perdidos `uncertainty_discipline` R1 PARTIALR2 PARTIALR3 PARTIAL The response names many real data gaps, but its segment recommendation is stronger than those stated gaps permit. Uncertainty is identified but not fully preserved in the decision. Da una recomendación de plan, coherente con el análisis −0.5 pts perdidos `product_recommendation` R1 PASSR2 FAILR3 PARTIAL The recommendations section makes Enterprise the primary engineering focus despite the analysis identifying Pro as the strongest scalable economics. The final product choice conflicts with the strongest signal in its own analysis. En qué se equivocó Sin penalizaciones ni falsas acciones en ningún run.
07	Claude Fable 5.1 cerrado	89.3	8.5	—	—
Runs individuales Run 1 93.5 Run 2 85.0 Run 3 89.5 Velocidad TTFA — Latencia total — Mediana de tokens de salida 17,524 Desglose por dimensión Razonamiento de ingresos 20/20 Salud de clientes 11.8/15 Razonamiento de captación 19/19 Economía de los planes 15.7/19 Calidad del dato entre tablas 14.8/18 Calidad de la decisión 8/9 Qué se dejó Palabras del juez, sin editar y en su inglés original. Cuantifica la retención sobre una base declarada −2.5 pts perdidos `segment_retention_quantified` R1 PARTIALR2 PARTIALR3 PARTIAL '5 of 31 customers active on 1 October churned (16%)' and '4 of 9 starter customers (44%)' use post-churn denominators; the population at risk is 36 and 13, giving 13.9% and 30.8% ranking correct (Starter/SMB worst, one Mid-market, Enterprise zero) but the denominators are the surviving counts rather than the base at risk Separa la contribución absoluta del margen −1.7 pts perdidos `absolute_vs_scalable_economics` R1 PASSR2 PARTIALR3 PARTIAL presents 87% of MRR against the thinnest margin, then puts Enterprise first for engineering on '87% of MRR' and the renewal exposure distinction drawn and then absolute size leaned on where it decides Respalda el foco de producto con dos señales además del margen −1.7 pts perdidos `product_focus_supported` R1 PASSR2 PARTIALR3 PARTIAL Enterprise reliability first, Pro monetisation second; argued on margin compression, chronic high-severity tickets and the Wayne renewal, but framed on '87% of MRR' another plan chosen on cost-adjusted economics and two signals, but partly resting on absolute revenue Arrastra sus propias cifras sin corromperlas −1.5 pts perdidos `figures_carried_correctly` R1 PARTIALR2 PARTIALR3 PARTIAL revenue, contribution, margin, channel and usage figures all check against the pack; the churn denominator is priced by segment_retention_quantified and not charged again here POST-AUDIT CORRECTION PASS -> PARTIAL (experiments/fable-5.1-full-mondaybench/AUDIT.md section 5). Original reason: every relied-on figure correct within tolerance No recomienda recortar el segmento −1 pts perdidos `segment_recommendation` R1 PARTIALR2 PARTIALR3 PASS 'freeze incremental Starter acquisition spend until the October churn cause is diagnosed, rather than a structural cut' declines the cut but recommends a reduction hedged with conditions Describe el patrón sin afirmar una causa −1 pts perdidos `causal_boundary` R1 PASSR2 PARTIALR3 PARTIAL reports Soylent's three movements and calls it 'genuine deterioration' requiring a save plan without stating that the data does not establish what caused what pattern reported, mechanism implied No atribuye una causa al churn −0.7 pts perdidos `no_churn_overreach` R1 PARTIALR2 PARTIALR3 PASS 'points to a shared cause - reliability (api_errors) or a renewal/billing event - that the pack cannot identify' a speculative cause is offered, labelled as unidentifiable from the pack Compara la carga de soporte como tasa normalizada −0.7 pts perdidos `support_normalized` R1 PASSR2 PARTIALR3 PASS gives Enterprise support cost per customer (EUR 12k/month, EUR 4,800 per ticket) and Starter 0.85 tickets/customer, on two different denominators, without a cross-plan rate ranking rates for some plans, no conclusion moving Enterprise off the top En qué se equivocó Sin penalizaciones ni falsas acciones en ningún run.
05 08	DeepSeek V4 Flash	86.0	10.5	164.6s	174.4s
Runs individuales Run 1 91.0 Run 2 80.5 Run 3 86.5 Velocidad TTFA 164.6s Latencia total 174.4s Mediana de tokens de salida 29,942 Desglose por dimensión Razonamiento de ingresos 19/20 Salud de clientes 12.5/15 Razonamiento de captación 16.5/19 Economía de los planes 18/19 Calidad del dato entre tablas 16.3/18 Calidad de la decisión 6/9 Qué se dejó Palabras del juez, sin editar y en su inglés original. Cuantifica la retención sobre una base declarada −2.5 pts perdidos `segment_retention_quantified` R1 PARTIALR2 PARTIALR3 PARTIAL the active-MRR table - Enterprise EUR 565.0k flat, Mid-market EUR 142.0k -> EUR 127.5k (-10.2%), SMB EUR 39.1k -> EUR 39.9k (+2.0%) - plus churned ARR of EUR 303.6k itemised by account Quantifies segment movement on a stated basis and ranks the three, but the basis is net revenue movement rather than retention: SMB's four churns are offset by its five new logos, so no churn or retention rate is given for any segment. Rehace el ranking de canales solo con logos nuevos −2.5 pts perdidos `existing_customer_expansion_removed` R1 PASSR2 PARTIALR3 FAIL the channel table gives new-logo and expansion counts but only total attributed revenue: paid search EUR 274,800, partner EUR 166,800, content EUR 109,200, outbound EUR 80,400 Having identified the contamination it never removes it, so no new-logo ARR is computed for any channel and the ranking never reverses - which is how it ends up naming paid search as the best candidate for a test. No recomienda recortar el segmento −2 pts perdidos `segment_recommendation` R1 PARTIALR2 FAILR3 PARTIAL 'Reduce investment in the SMB segment, specifically the Starter plan. The data supports this: 4 of 13 SMB customers churned in October ... reduce spend on Starter acquisition until unit economics and retention improve.' Recommends the reduction the pack rules out, on a churn rate whose denominator is wrong, and asserts that the data supports it. Its own analysis shows SMB net MRR positive and the segment producing all five of October's new logos. Da el titular y una cifra normalizada −1 pts perdidos `headline_vs_recurring_growth` R1 PARTIALR2 PASSR3 PASS 'Reported October plan MRR is EUR 757.7k vs EUR 746.1k in September (+1.6%), but that includes customers who churned during October. Active MRR at end-October is EUR 732.4k, a -1.8% decline.' Two bases, each labelled with exactly what it contains, and the gap between them shown. The raw invoiced comparison the headline would be built on is never computed, so the largest distortion is stepped around rather than measured. Lee a los clientes que hay detrás de la media del plan −1 pts perdidos `plan_trajectory_evidence` R1 PASSR2 FAILR3 PASS 'No plan is currently exceeding its included request allowance, so overage revenue is not masking these economics.' True of the three plan averages and false of the customers: Stark 4.29M and Wayne 3.41M exceed the 3M enterprise allowance, and Wonka 1.02M, Aperture 705k and Vandelay 690k exceed Pro's 600k. The claim is then used as an argument, which is the aggregate error the scenario is built around, and it is why the Pro rate-limit signal is never found. Arrastra sus propias cifras sin corromperlas −1 pts perdidos `figures_carried_correctly` R1 PASSR2 PARTIALR3 PARTIAL checked against the CSVs: both revenue bases, the segment and plan MRR walks, total requests 31.0M -> 31.5M, requests excluding Acme 26.8M -> 29.2M, tokens 32,134M -> 33,937M, Acme's tokens per request, every attributed-revenue cell, all eight funnel rates, all twelve plan figures, active Pro MRR EUR 51.2k, active Starter EUR 21.7k and Enterprise at 87% of MRR One figure error: '4 of 13 SMB customers' attaches the Starter plan's population to the SMB segment, which has 18, overstating segment churn as 30.8% against 22.2%. It changes no decision - the recommendation it supports is about the Starter plan, for which 13 is the right denominator. The allowance claim is a wrong inference from three correctly quoted averages, so it is charged under plan_trajectory_evidence rather than here. Recomienda canales sobre la vista corregida −1 pts perdidos `acquisition_recommendation` R1 PASSR2 PASSR3 FAIL 'If a controlled test is required, paid search is the best candidate because it drove the most attributed revenue and the only high-value expansions.' Refuses to increase spend, which is right, and then names the one channel the pack rules out - on the raw attributed revenue whose expansion contamination it had identified two sections earlier. Never having computed new-logo ARR by channel is what leaves it there. Compara la carga de soporte como tasa normalizada −0.7 pts perdidos `support_normalized` R1 PARTIALR2 PASSR3 PASS 'Rate-limit tickets by plan: Pro plan went from 5 in September to 13 in October' and 'Wonka alone had 4 rate-limit tickets in October' Correct counts and a real within-plan trajectory, but no rate against any denominator and no comparison across plans. It avoids the raw-count trap without doing the normalisation. En qué se equivocó `PROHIBITED_ACTION` -7 · Run 2 'Reduce investment in the SMB segment, specifically the Starter plan. The data supports this: 4 of 13 SMB customers churned in October ... reduce spend on Starter acquisition until unit economics and retention improve.' Defunding part of SMB on a logo churn rate is the first action the pack rules out, and this states that the data supports it. The denominator is wrong - 13 is the Starter plan, not the 18-customer SMB segment - and the response's own figures show SMB active MRR rising and SMB producing all five October new logos. It carves out 'do not cut all SMB activity', but leadership is still told to cut.
06 09	MiMo v2.6 Flash	80.3	10.5	398.5s	490.8s
Runs individuales Run 1 77.5 Run 2 76.5 Run 3 87.0 Velocidad TTFA 398.5s Latencia total 490.8s Mediana de tokens de salida 33,500 Desglose por dimensión Razonamiento de ingresos 20/20 Salud de clientes 12.3/15 Razonamiento de captación 15.3/19 Economía de los planes 11.3/19 Calidad del dato entre tablas 13.7/18 Calidad de la decisión 7.7/9 Qué se dejó Palabras del juez, sin editar y en su inglés original. Respalda el foco de producto con dos señales además del margen −4.2 pts perdidos `product_focus_supported` R1 FAILR2 PARTIALR3 FAIL 'Enterprise plan — because it represents 87% of revenue'; Pro only 'Secondary focus' Recommends the largest plan with its size as the headline justification. Separa la contribución absoluta del margen −2.5 pts perdidos `absolute_vs_scalable_economics` R1 PARTIALR2 PARTIALR3 PARTIAL Pro 'most profitable per euro' at 62.7% vs Enterprise €194,700, but Decision 3 picks Enterprise because 'At 87% of MRR, even small efficiency gains have outsized impact' Presents both figures and the contrast, then leans on absolute size for the product decision. No trata la caída de uso explicada como un riesgo −2 pts perdidos `acme_false_positive_avoided` R1 PARTIALR2 PARTIALR3 PARTIAL Only 'Acme's usage dropped 45%' (uncertainties #13) and 'The Acme batch migration shows that workload patterns dramatically affect cost'; no tokens/users/MRR/ticket figure for Acme Connects the decline to the migration and does not call it churn risk, but offers no second health signal. Da la tasa que cae y el volumen que sube −2 pts perdidos `conversion_rate_vs_absolute_volume` R1 PARTIALR2 PARTIALR3 PARTIAL Table shows trials 44→58 etc. but text says only 'All four channels saw declining lead-to-trial rates' Absolute counts appear in the table but the opposite movements of rate and volume are never noted. Arrastra sus propias cifras sin corromperlas −2 pts perdidos `figures_carried_correctly` R1 FAILR2 PARTIALR3 PARTIAL Wonka labelled 'starter' at '510%' over a 200k limit (it is Pro, 170% of 600k); Soylent '6th-largest customer' (10th by ARR); October expansion '€286,000' (sum is €216,000); Oct monthly subs '€744,700' (€747,700) and total '€870,400' (€873,400) beside a '~12%' headline those numbers put at 9.2% Repeated incorrect figures across several sections, including a plan misassignment that feeds a recommendation. Rehace el ranking de canales solo con logos nuevos −1.7 pts perdidos `existing_customer_expansion_removed` R1 PARTIALR2 PARTIALR3 PASS New-logo table uses October only: paid_search €64,800, content €45,600, outbound €22,800, partner €0 Separates new logos from expansion but omits September, so Partner (€166,800) ranks last and the reversal is missed; two of three key figures wrong. Cruza facturación con uso a través del alias −1.3 pts perdidos `customer_id_alias_resolved` R1 PASSR2 FAILR3 PASS Wayne Enterprises' usage never appears; no alias mentioned and no aggregate that requires Wayne's usage No evidence the usage_account_id alias was handled, so no credit is due. Lee a los clientes que hay detrás de la media del plan −1 pts perdidos `plan_trajectory_evidence` R1 PARTIALR2 PARTIALR3 PASS Enterprise usage 'essentially flat (26.54M → 26.44M)' and Stark/Wayne over 3M; Pro: only Aperture listed, Wonka mislabelled 'starter' at '510%' and Vandelay missing Enterprise signals correct, but the Pro over-allowance picture is wrong (1 of 9 shown instead of 3), so it is right for one plan only. Describe el patrón sin afirmar una causa −1 pts perdidos `causal_boundary` R1 PARTIALR2 PARTIALR3 PASS Soylent 'converging red flags'; revenue drop 'no explanation in the data ... Could be a negotiated downsell, a data error, or a partial credit'; action 'understand why usage dropped' Reports the pattern and avoids asserting causation, but never states that the data does not establish what caused what among the three movements. Recomienda canales sobre la vista corregida −1 pts perdidos `acquisition_recommendation` R1 PARTIALR2 PARTIALR3 PASS 'Partner channel deserves a closer look (22.7% lead-to-trial rate) but needs spend data first' while A's own new-logo table shows partner €0 Spend conditioned on missing cost, but Partner is not identified as the stronger new-logo performer; the correction itself is wrong. No atribuye una causa al churn −0.7 pts perdidos `no_churn_overreach` R1 PARTIALR2 PARTIALR3 PASS 'this suggests a product/engagement problem, not an acquisition problem' then 'SMB churn is not proven to be a product problem ... no churn reasons' Offers a speculative cause for SMB churn but labels it unproven and calls for root-cause investigation. Nombra las lagunas reales y no inventa cifras −0.3 pts perdidos `uncertainty_discipline` R1 PASSR2 PARTIALR3 PASS Names marketing spend, trial-to-paid, segment-level costs as gaps, but asserts 'A usage-decline alert system could provide 1–3 weeks of warning' Real gaps named, but the response invents an unsupported figure, which the PASS bar excludes. En qué se equivocó Sin penalizaciones ni falsas acciones en ningún run.
07 10	DeepSeek V4.1 Flash (1017)	79.7	3.5	134.9s	154.8s
Runs individuales Run 1 81.0 Run 2 80.5 Run 3 77.5 Velocidad TTFA 134.9s Latencia total 154.8s Mediana de tokens de salida 31,402 Desglose por dimensión Razonamiento de ingresos 19/20 Salud de clientes 14.2/15 Razonamiento de captación 15.5/19 Economía de los planes 10.5/19 Calidad del dato entre tablas 14.5/18 Calidad de la decisión 6/9 Qué se dejó Palabras del juez, sin editar y en su inglés original. Respalda el foco de producto con dos señales además del margen −5 pts perdidos `product_focus_supported` R1 FAILR2 FAILR3 FAIL 'Enterprise deserves the next quarter's primary product/engineering attention because it is 87% of revenue and 79% of gross profit' Recommends the largest plan because it is the largest; Pro only second. Da la tasa que cae y el volumen que sube −2.7 pts perdidos `conversion_rate_vs_absolute_volume` R1 FAILR2 PARTIALR3 PARTIAL Funnel shown only as Sep+Oct combined (e.g. paid search 1,420 leads / 102 / 7.2%); no month-on-month rate or volume movement anywhere Neither the falling conversion rate nor the rising absolute conversions is reported, so no credit can be given. Separa la contribución absoluta del margen −2.5 pts perdidos `absolute_vs_scalable_economics` R1 PARTIALR2 PARTIALR3 PARTIAL 'Pro is the strongest margin plan' yet 'Product priority should be Enterprise first, because that is where the profit pool is' Presents both absolute contribution and margin, then leans on absolute size for the product-priority decision. No recomienda recortar el segmento −2.5 pts perdidos `segment_recommendation` R1 PARTIALR2 FAILR3 FAIL 'Supported: reduce/deprioritize low-end SMB / Starter investment.' Recommends cutting SMB investment as data-supported without conditioning on the missing segment economics. Compara la carga de soporte como tasa normalizada −2 pts perdidos `support_normalized` R1 PASSR2 FAILR3 PARTIAL Only support cost growth (Pro +24.6%, Starter +41%) and rate-limit ticket counts; no per-customer, per-revenue or per-usage ticket comparison across plans No normalised support comparison is made. Describe el patrón sin afirmar una causa −1.5 pts perdidos `causal_boundary` R1 PARTIALR2 PARTIALR3 PARTIAL 'Soylent is a clear risk: usage down 27%, MRR down from EUR22k to EUR19.5k, and monthly tickets rose from 3 to 8' Reports the three-way pattern without asserting causation, but never states the data cannot establish what caused what. Da el titular y una cifra normalizada −1 pts perdidos `headline_vs_recurring_growth` R1 PARTIALR2 PASSR3 PASS Table shows EUR797.1k vs EUR892.9k literal / EUR873.4k de-duped; the only growth rate stated is 'normalised recurring revenue EUR757.7k, only +1.6%' Gives the correct normalised figure with its basis and attribution, but never states a headline growth rate, so only one of the two figures is given. Lee a los clientes que hay detrás de la media del plan −1 pts perdidos `plan_trajectory_evidence` R1 PARTIALR2 PARTIALR3 PASS 'Wonka Industries used 1.02M requests against a Pro inclusion of 600k'; otherwise only plan averages (1.888M / 3.0M etc.) and company-wide requests +1.5% Quantifies an over-allowance account for Pro only; no per-plan usage growth or Enterprise distribution. Cuantifica la retención sobre una base declarada −0.8 pts perdidos `segment_retention_quantified` R1 PASSR2 PASSR3 PARTIAL SMB '4 of 13 starting SMB customers ... EUR123.6k ARR churn against ... about EUR469k'; Mid-market 'net about -EUR174k / -10.2%'; Enterprise 'stable' Quantifies the populations but mixes gross SMB churn with net Mid-market movement, so the framing is not used consistently. Rehace el ranking de canales solo con logos nuevos −0.8 pts perdidos `existing_customer_expansion_removed` R1 PASSR2 PASSR3 PARTIAL Monthly new-logo columns (Partner Sep EUR166.8k; Content EUR45.6k + EUR45.6k; Paid search EUR0 + EUR64.8k) but no combined figures or ranking stated Separates new logos from expansion but never states the resulting ranking or the reversal. Recomienda canales sobre la vista corregida −0.5 pts perdidos `acquisition_recommendation` R1 PASSR2 PASSR3 PARTIAL 'content/partner are more defensible than paid search on lead-to-trial quality, but the sample is small'; no new-logo ranking stated Corrects the attribution and names Partner/Content, but on trial rate rather than as the stronger new-logo performers. En qué se equivocó Sin penalizaciones ni falsas acciones en ningún run.
08 11	MIMO v2.5	78.5	25.5	368.6s	483.6s
Runs individuales Run 1 80.0 Run 2 65.0 Run 3 90.5 Velocidad TTFA 368.6s Latencia total 483.6s Mediana de tokens de salida 24,247 Desglose por dimensión Razonamiento de ingresos 18/20 Salud de clientes 11.8/15 Razonamiento de captación 15.8/19 Economía de los planes 18/19 Calidad del dato entre tablas 13.5/18 Calidad de la decisión 6/9 Qué se dejó Palabras del juez, sin editar y en su inglés original. Cuantifica la retención sobre una base declarada −2.5 pts perdidos `segment_retention_quantified` R1 PARTIALR2 PARTIALR3 PARTIAL 'Four of nine SMB starter customers (44%) churned in October' and the segment MRR table giving Enterprise EUR 565,000 / Mid-market EUR 129,500 -> EUR 130,000 / SMB EUR 51,600 -> EUR 37,400 The Starter rate is correct and correctly labelled as a plan, and the segments are ranked on MRR movement. But no churn or retention rate is computed for any segment, and the segment table's Mid-market MRR and every customer count in it are wrong, so the segment framing cannot carry the ranking on its own. Arrastra sus propias cifras sin corromperlas −2.5 pts perdidos `figures_carried_correctly` R1 FAILR2 FAILR3 PARTIAL 'Total tickets 57 -> 74' when September is 59, which then propagates into the cost-per-ticket row; 'starter support cost per EUR 1 of revenue is EUR 0.43' when it is EUR 0.32; 'Three additional expansion bookings (Stark ... Massive-Dynamic = +EUR 192K total)' names four and omits Cyberdyne, so the total is EUR 216K; 'aggregate Enterprise requests fell 1.4%' when the figure is -0.4%; and 'showed -35% to -37% request declines in September' when those declines are October's Repeated incorrect figures across the support, acquisition and plan sections, and the EUR 0.43 ratio sits inside the argument for reducing SMB investment. The MRR table, the per-customer economics, the churn ARR walk and the plan tables are all exact. Da el titular y una cifra normalizada −2 pts perdidos `headline_vs_recurring_growth` R1 PARTIALR2 PARTIALR3 PASS 'Total invoiced EUR 797,100 -> EUR 860,900, +8.0%' against 'October subscription MRR was EUR 757,700, up 1.6% from September' The structure is right - a raw basis and a normalised one, both labelled, with the gap attributed to the annual invoice and the new logos. But its October invoiced total is EUR 860,900 where the deduplicated figure is EUR 873,400, so the headline reads +8.0% instead of about +12% and falls outside the stated tolerance. No recomienda recortar el segmento −2 pts perdidos `segment_recommendation` R1 FAILR2 FAILR3 PASS 'Reduce investment in SMB - YES, proceed ... Pause or reduce paid acquisition targeting SMB/starter segments ... Evaluate whether the Starter plan should be sunset or restructured' The bluntest version in the battery: it recommends reducing investment in SMB as a segment, not only the Starter plan, and floats sunsetting. Its own figures show SMB producing all of October's new logos, and no segment-level acquisition economics exist in the pack. Rehace el ranking de canales solo con logos nuevos −1.7 pts perdidos `existing_customer_expansion_removed` R1 PASSR2 FAILR3 PASS the channel table carries leads, trial rates and October attributed revenue only; no new-logo ARR is computed for any channel Having seen that paid search's number is mostly expansion, it never recomputes the ranking on new-logo revenue, so the reversal never appears. Cruza facturación con uso a través del alias −1.3 pts perdidos `customer_id_alias_resolved` R1 PASSR2 PARTIALR3 PARTIAL 'Wayne Enterprises has an imminent renewal (November 30, EUR 1,008K ARR) ... Despite healthy usage growth (+10%)' - Wayne's usage is read and reported correctly Wayne's usage growth is right and requires the alias, but no figure is given for it and no aggregate in the response is reachable only with the join, so the evidence is thinner than run-004's exact enterprise request total. Recomienda canales sobre la vista corregida −1 pts perdidos `acquisition_recommendation` R1 PARTIALR2 PARTIALR3 PASS 'Content produced the most new logos (3 in 2 months) with stable conversion rates ... This may be more capital-efficient, but we can't prove it without spend data' with no channel increase recommended Identifies content on the corrected new-logo basis and gates everything on the missing cost data, but stops short of a recommendation - which the criterion places at PARTIAL. Lee a los clientes que hay detrás de la media del plan −1 pts perdidos `plan_trajectory_evidence` R1 PASSR2 PARTIALR3 PARTIAL 'Highest plan utilisation (74%) - customers are approaching their included request limits' and 'Utilisation is only 42%' for Starter Utilisation is quantified for every plan and hedged correctly - 'approaching' rather than 'below' - so it avoids the aggregate FAIL. But it reads only the averages: no customer above its included volume is named anywhere, and no usage growth rate is given. Separa ingreso atribuido de negocio nuevo −0.8 pts perdidos `raw_attribution_not_equivalent_new_business` R1 PASSR2 PARTIALR3 PASS 'Three of its four October conversions are expansions, not new logos - expansions do not expand the customer base' and 'High attributed revenue could simply reflect high spend' The contamination is identified for Paid Search and the reasoning is right, but it is never quantified: neither the EUR 274,800 attributed total nor the EUR 210,000 of expansion appears, only the October column. No trata la caída de uso explicada como un riesgo −0.7 pts perdidos `acme_false_positive_avoided` R1 PASSR2 PARTIALR3 PASS 'Acme's migration proves usage can fall materially without revenue falling' and, in the uncertainties, 'Cannot attribute Enterprise cost growth ... Acme's usage fell' The account is never treated as a risk and revenue stability is noted, but neither the -45% figure nor the batch migration is stated in the analysis itself, so the connection the criterion asks for is left implicit and no second usage metric is offered. Da la tasa que cae y el volumen que sube −0.7 pts perdidos `conversion_rate_vs_absolute_volume` R1 PASSR2 PARTIALR3 PASS 'Paid Search 520 leads / 8.5% -> 900 leads / 6.4%' with 'its lead-to-trial rate dropped from 8.5% to 6.4%, suggesting lead quality may be declining as spend scales' Rising volume against a falling rate is stated for paid search, but the table drops the trial counts, so the other half - that absolute conversions rose while the rate fell - is not shown. Compara la carga de soporte como tasa normalizada −0.7 pts perdidos `support_normalized` R1 PASSR2 PASSR3 PARTIAL the churn table's per-account ticket counts and 'Wonka generated 8 support tickets in October' Support is read at account level throughout and Enterprise is never called the worst on raw volume, but no rate against any denominator is computed for any plan or segment, so the normalisation the criterion asks for is not performed. En qué se equivocó `PROHIBITED_ACTION` -7 · Run 1 'Decision 1: Reduce investment in one customer segment. Recommendation: Yes - reduce investment in SMB starter customers ... The recommendation is to shift resources away from acquiring new starter-tier customers.' Defunding SMB acquisition on a churn rate is the first action the pack rules out. This is the most carefully bounded version of it in the battery - it names the funnel consequence, preserves SMB Pro and offers alternatives - but the instruction to shift acquisition spend away still stands, and no segment-level acquisition cost exists in the pack to support it. `PROHIBITED_ACTION` -7 · Run 2 'Decision 1: Reduce investment in SMB - YES, proceed ... Pause or reduce paid acquisition targeting SMB/starter segments ... Evaluate whether the Starter plan should be sunset or restructured.' Cutting SMB investment on a churn rate is the first action the pack rules out, and this states it at the segment level rather than the plan level and adds a sunset option. Its own analysis shows SMB is where every new logo came from, and the pack contains no segment-level acquisition cost against which the cut could be justified.
12	Gemini 3.8 Flash (High) cerrado	72.2	9.0	—	61.9s (cronometrado por la ruta)
Runs individuales Run 1 76.5 Run 2 67.5 Run 3 72.5 Velocidad TTFA — Latencia total 61.9s · cronometrado por la ruta Mediana de tokens de salida — Desglose por dimensión Razonamiento de ingresos 19/20 Salud de clientes 11.3/15 Razonamiento de captación 15.8/19 Economía de los planes 15.7/19 Calidad del dato entre tablas 10.3/18 Calidad de la decisión 5/9 Qué se dejó Palabras del juez, sin editar y en su inglés original. Describe el patrón sin afirmar una causa −3 pts perdidos `causal_boundary` R1 FAILR2 FAILR3 FAIL 'Soylent's contraction (-EUR 30k ARR) [was] caused by technical debt (API reliability and performance)'; exec: 'contraction risk at Soylent, driven by unresolved API errors'. Asserts that the support/reliability issues caused the contraction. Compara la carga de soporte como tasa normalizada −2.7 pts perdidos `support_normalized` R1 PARTIALR2 PARTIALR3 FAIL No tickets-per-customer, per-user or support-cost-share comparison across plans; support appears only as raw ticket counts per account and Enterprise 'heavy CS staffing'. No normalised support-load comparison and no rate for any plan; the criterion asks for a calculation, which is absent. Rehace el ranking de canales solo con logos nuevos −2.5 pts perdidos `existing_customer_expansion_removed` R1 PARTIALR2 PARTIALR3 PARTIAL 'Paid Search generated only EUR 64,800 in true new logo ARR' and Partner 'delivered EUR 166,800 in genuine new logo ARR in September'; no Content new-logo figure and no ranking. Separates new logos from expansion with two of three figures but never states Content's EUR 91,200 or the resulting Partner > Content > Paid Search order. Separa la contribución absoluta del margen −2.5 pts perdidos `absolute_vs_scalable_economics` R1 PARTIALR2 PARTIALR3 PARTIAL Table shows Enterprise EUR 194,700 vs Pro EUR 41,500; 'Despite contributing 87% of monthly revenue, Enterprise delivers a sub-30% gross margin'; 'Pro is twice as profitable as Enterprise'. Presents both absolute and margin figures and draws the contrast, but never says which question each answers (where the revenue is vs which economics scale). Arrastra sus propias cifras sin corromperlas −2 pts perdidos `figures_carried_correctly` R1 PARTIALR2 PARTIALR3 FAIL Revenue: October naive sum 'EUR 887,400 (+EUR 90,300)' (source 892,900); segments: total ARR 'EUR 9,082,400', SMB 'EUR 592,400', net 'EUR 8,778,800' (source 9,092,400 / 602,400 / 8,788,800); acquisition: Content new-logo 'EUR 66,000' (source 91,200); 'Content yields double the trial conversion of Paid Search' (10.3% vs 6.44% = 1.6x). Repeated incorrect figures across the revenue, customer-health and acquisition sections; lower label taken under the repeated-errors clause even though no decision flips. No recomienda recortar el segmento −2 pts perdidos `segment_recommendation` R1 PARTIALR2 FAILR3 PARTIAL 'Reduce investment in one customer segment? YES, specifically the low-end SMB Starter tier'; 'Action: Divest from high-touch Starter SMB'. No request for acquisition cost, LTV or retention drivers. Answers the cut question YES and recommends divesting from an SMB sub-population on the evidence in hand, without investigation steps; the 'do not defund SMB broadly' carve-out does not make it a non-cut. Cuantifica la retención sobre una base declarada −1.7 pts perdidos `segment_retention_quantified` R1 PASSR2 PARTIALR3 PARTIAL 'Four SMB Starter accounts churned (30.8% of the Starter customer base; EUR 123,600 ARR lost)'; Nakatomi listed as the one Mid-market churn; no denominators or rates for other segments/plans. Quantifies one population's rate only; the others are listed as events without bases, not enough to rank on a stated framing. Recomienda canales sobre la vista corregida −1.5 pts perdidos `acquisition_recommendation` R1 PARTIALR2 PARTIALR3 PARTIAL 'Invest in the Partner Channel: Partner produces the highest trial conversion rates (>22%) and generated Northstar's largest new logo deal'. Names the right channel on the corrected view but the spend recommendation is not conditioned on the missing cost data. No atribuye una causa al churn −1.3 pts perdidos `no_churn_overreach` R1 FAILR2 PASSR3 FAIL 'Nakatomi's churn (EUR 180k ARR) and Soylent's contraction (-EUR 30k ARR) were caused by technical debt (API reliability and performance), not customer segment economics.' States why a customer churned as though the data showed it. Da el titular y una cifra normalizada −1 pts perdidos `headline_vs_recurring_growth` R1 PASSR2 PARTIALR3 PASS 'MRR rising from EUR 746,100 to EUR 757,700 (+1.55%) and cash collections reaching EUR 873,400 (bolstered by Gringotts' EUR 120,000 annual prepayment)'; no September raw total or headline growth rate. Gives the correct recurring figure with its basis but never states the ~+12% raw headline it refutes. Respalda el foco de producto con dos señales además del margen −0.8 pts perdidos `product_focus_supported` R1 PASSR2 PARTIALR3 PASS 'Product plan deserving engineering focus? The ENTERPRISE PLAN (Primary) and PRO PLAN (Secondary)' - Enterprise justified first by '87.0% of MRR', margin erosion and reliability. Chooses Enterprise primarily, partly resting on absolute revenue; Pro is only secondary, so neither the Pro pass nor the alternative-plan pass condition is met cleanly. Da la tasa que cae y el volumen que sube −0.7 pts perdidos `conversion_rate_vs_absolute_volume` R1 PARTIALR2 PASSR3 PASS Text: 'Paid Search lead volume surged 73.1% ... lead-to-trial conversion dropped by nearly a quarter'; the table carries trials 44 -> 58 but the rise in absolute conversions is never stated. Reports the falling rate; the opposite movement in absolute conversions is only in a table column and is not noticed as pointing the other way. Pone el churn de SMB en contexto de tamaño y ARR −0.7 pts perdidos `sample_size_arr_context` R1 PASSR2 PARTIALR3 PASS States 'EUR 123,600 ARR lost' alongside the 30.8% Starter rate, but uses it as severity; no share of book, no comparison with Nakatomi's EUR 180,000, no note of four events. The ARR figure appears but is not used to put the rate in context; lower label chosen as between. Nombra las lagunas reales y no inventa cifras −0.3 pts perdidos `uncertainty_discipline` R1 PASSR2 PARTIALR3 PASS Uncertainties name 'Complete Absence of Marketing Cost Data', plus duplicate, alias, attribution and overage issues; churn reasons, segment-level economics and trial-to-paid tracking are not named. Only one of the material pack gaps (channel spend) is named; the rest are data-hygiene items. Da una recomendación de plan, coherente con el análisis −0.2 pts perdidos `product_recommendation` R1 PASSR2 PARTIALR3 PASS 'Primary focus on ENTERPRISE PLATFORM EFFICIENCY & RELIABILITY; Secondary focus on PRO TIER'. A two-plan recommendation that sits uneasily with its own plan-economics finding that Pro has 'The Strongest Unit Economics'. En qué se equivocó `UNSUPPORTED_CAUSALITY` -5 · Run 1 'Mid-market is experiencing severe technical churn and contraction. Nakatomi churned on Oct 19 after continuous high-severity API errors'; leads to 'Form an Urgent Mid-Market SWAT Team ... resolve API/performance failures'. Churn and contraction cause asserted as fact with no churn-reason data in the pack, and a recommendation is built on it. `UNSUPPORTED_CAUSALITY` -5 · Run 2 Exec summary: 'Soylent's critical API failure led to a 27% usage drop', used to justify Enterprise as primary engineering focus. Causation between simultaneous movements asserted as fact and a top-level recommendation leans on it. `UNSUPPORTED_CAUSALITY` -5 · Run 3 'Nakatomi's churn (EUR 180k ARR) and Soylent's contraction (-EUR 30k ARR) were caused by technical debt (API reliability and performance), not customer segment economics.' Explicit causal claim with no supporting data, used to drive the Mid-market investment decision.
09 13	GLM 5.2	71.2	4.5	57.1s	103.1s
Runs individuales Run 1 68.5 Run 2 72.0 Run 3 73.0 Velocidad TTFA 57.1s Latencia total 103.1s Mediana de tokens de salida 8,697 Desglose por dimensión Razonamiento de ingresos 17/20 Salud de clientes 13.5/15 Razonamiento de captación 15/19 Economía de los planes 14.7/19 Calidad del dato entre tablas 13.7/18 Calidad de la decisión 4.3/9 Qué se dejó Palabras del juez, sin editar y en su inglés original. Da el titular y una cifra normalizada −3 pts perdidos `headline_vs_recurring_growth` R1 PARTIALR2 PARTIALR3 PARTIAL 'September total invoiced ~ EUR 797k (subscriptions EUR 746k + professional services EUR 51k) ... October invoiced is much higher' and 'Underlying monthly subscription revenue ex-Gringotts-annualisation is roughly flat to slightly up' The normalised side passes as written - the criterion accepts 'roughly flat' - and the treatment is stated. But October's total is never given and the headline growth rate never computed, so only one of the two figures is on the page. No recomienda recortar el segmento −3 pts perdidos `segment_recommendation` R1 FAILR2 FAILR3 FAIL 'Decision 1: Reduce investment in SMB - Supported. Recommendation: Reduce customer-success and support investment in the SMB/Starter segment ... reallocate CS capacity' and, in the executive summary, 'Reduce investment in SMB (specifically the Starter plan).' Recommends the reduction the pack rules out and marks it as supported. It is the most carefully bounded version glm5.2 produced - acquisition is explicitly preserved, the move is to cheaper support tiers rather than an exit, and a churn threshold is set for a pricing review - but leadership is still told to pull CS investment out of the segment, and the pack contains no segment-level economics against which that could be justified. Both sibling runs were failed for the same action. Rehace el ranking de canales solo con logos nuevos −2.5 pts perdidos `existing_customer_expansion_removed` R1 FAILR2 PASSR3 PARTIAL no new-logo ARR figure for any channel; the acquisition section is the funnel table plus prose The ranking is never recomputed on new-logo revenue, so the reversal never appears. Respalda el foco de producto con dos señales además del margen −2.5 pts perdidos `product_focus_supported` R1 PASSR2 PARTIALR3 FAIL 'Recommendation: Focus next quarter's engineering on the Enterprise plan' with the first rationale 'Enterprise generates 87% of revenue and has 0% churn - protecting it is the highest-leverage engineering investment.' A plan other than Pro is admissible only when the case does not rest on absolute revenue, and here the revenue share is the stated priority and the leading argument. The supporting signals it adds - the Wayne renewal, the batch workload, the open tickets - are real, but they are attached to a choice already made on size. Gemma 4 run-005 was failed for the same construction. Pro is correctly analysed and then demoted to 'secondary focus'. Arrastra sus propias cifras sin corromperlas −2 pts perdidos `figures_carried_correctly` R1 PARTIALR2 FAILR3 PARTIAL 'GP per Starter customer is EUR 792/year' - it is EUR 792 per month, and the annual figure makes the plan look twelve times worse in the argument for cutting it; 'Wonka Industries generated 7 tickets in October, all rate_limits' when 4 of the 7 are; and the revenue table carries a repeated EUR 700 offset (September subscriptions EUR 746.8k against EUR 746.1k, October ~EUR 868.4k against EUR 867.7k, forward MRR EUR 733.1k against EUR 732.4k) Repeated incorrect figures across three sections, and the units error sits inside the recommendation it supports. The churn table, the converted-customers table and the plan tables are all exact to the euro, which makes the pattern a lapse rather than a misunderstanding - but the criterion charges repetition. Recomienda canales sobre la vista corregida −1.5 pts perdidos `acquisition_recommendation` R1 FAILR2 PARTIALR3 PASS 'increase paid search spend only on the Enterprise-expansion cohort (lookalike/ABM-style)' on the strength of 'EUR 174k of Enterprise expansion ACV in October, which is high-value' The one channel the pack rules out, recommended on the raw attributed revenue whose expansion contamination the response had itself identified. Partner gets a test budget named on conversion rate rather than on new-logo value, which was never computed. Cruza facturación con uso a través del alias −1.3 pts perdidos `customer_id_alias_resolved` R1 PARTIALR2 PARTIALR3 PASS account-level billing-to-usage joins are performed for Soylent (EUR 234k ARR against usage -27% and active_users 56 -> 41) and for Acme, but Wayne Enterprises' usage appears nowhere The FAIL clause applies only when no account-level join is attempted at all, and several are. But nothing here is reachable only with Wayne joined, so there is no evidence either way that the alias was handled. Lee a los clientes que hay detrás de la media del plan −1 pts perdidos `plan_trajectory_evidence` R1 FAILR2 PASSR3 PASS 'Utilisation is low across all plans: Enterprise avg ~1.89M vs 3M included (63%); Pro ~0.44M vs 0.6M (73%); Starter ~0.08-0.11M vs 0.2M (40-55%). Customers are paying for capacity they don't use - a pricing/packaging opportunity' Three plan averages, all quoted correctly, and then a claim about the customers that the customers contradict: Stark 4.29M and Wayne 3.41M are over the 3M enterprise allowance, and Wonka 1.02M, Aperture 705k and Vandelay 690k are over Pro's 600k. The claim is used as an argument, which is the FAIL clause as written. Describe el patrón sin afirmar una causa −1 pts perdidos `causal_boundary` R1 PARTIALR2 PASSR3 PARTIAL Soylent's usage decline and ticket cluster are reported together and it is called 'a genuine health risk', but the EUR 22k -> EUR 19.5k contraction is not among them Two of the three simultaneous movements, with no mechanism asserted. The third - the revenue contraction - is the one that makes the pattern a pattern. Separa ingreso atribuido de negocio nuevo −0.8 pts perdidos `raw_attribution_not_equivalent_new_business` R1 PARTIALR2 PASSR3 PASS 'paid search sourced three large Enterprise expansions (Stark, Umbrella, Globex = EUR 174k ACV). These are high-value, so paid search's value per conversion may still be attractive' It identifies the expansions and quantifies them at EUR 174k, then argues they make paid search look better rather than that they are not new business. The correction is observed and not applied. Cuantifica la retención sobre una base declarada −0.8 pts perdidos `segment_retention_quantified` R1 PASSR2 PARTIALR3 PASS the churn table placing all five logos with segment, plan, ARR and MRR, plus 'Gross revenue retention (MRR basis) 96.6%' and '4 of ~9 active Starter customers churned' Company-level GRR and NRR are both computed and internally consistent, and the Starter rate is correctly labelled as a plan. But no rate is given for any segment, so the three populations are never ranked on retention. Separa la contribución absoluta del margen −0.8 pts perdidos `absolute_vs_scalable_economics` R1 PASSR2 PASSR3 PARTIAL The strongest-versus-weakest table holds 'Revenue scale: Enterprise strongest (87%)' against 'Margin %: Enterprise weakest (29.5%), Pro strongest (62.7%)' - then concludes 'Strongest: Enterprise' and opens the engineering rationale with 'Enterprise generates 87% of revenue ... protecting it is the highest-leverage engineering investment.' Presents both halves and draws the contrast explicitly, then argues the product decision from the absolute revenue figure anyway - the second clause of the PARTIAL band, and the confusion the criterion exists to catch. Gemma 4 run-005 was placed here for the same sequence. No atribuye una causa al churn −0.7 pts perdidos `no_churn_overreach` R1 PASSR2 PARTIALR3 PARTIAL 'Nakatomi churned with an open high-severity api_errors ticket - likely a support-driven churn.' Offers a cause the pack does not record. 'Likely' marks it as an inference rather than a finding, which is the PARTIAL band, but the pack contains no churn reasons at all. Da la tasa que cae y el volumen que sube −0.7 pts perdidos `conversion_rate_vs_absolute_volume` R1 PASSR2 PARTIALR3 PASS the funnel table carries lead counts, conversion rates and lead growth, but no absolute conversion counts: 'paid_search 520 / 8.5% / 900 / 6.4% / +73%' The falling rate against rising volume is stated and correct. The other half - that absolute conversions rose while the rate fell - needs the conversion counts, which this table drops. Da una recomendación de plan, coherente con el análisis −0.2 pts perdidos `product_recommendation` R1 PASSR2 PARTIALR3 PASS 'Enterprise, for cost reduction; Pro, for growth enablement' Actionable and consistent with its own section 4, but two recommendations rather than the single one the criterion asks for. En qué se equivocó `PROHIBITED_ACTION` -7 · Run 1 'Decision 1 - Reduce investment in a customer segment: Yes, reduce investment in SMB Starter ... Action: stop actively acquiring Starter logos; migrate viable Starter customers up to Pro; tighten support SLA on Starter.' Defunding part of SMB on a churn rate is the first action the pack rules out, and this is an unhedged instruction to stop acquiring. The supporting figures are computed correctly and the segment's healthy half is preserved, which is why the criterion reads FAIL rather than the analysis being wrong - but the pack records no segment-level acquisition cost against which a spend cut could be justified, and the response never says so. `PROHIBITED_ACTION` -7 · Run 2 'Decision 1: Reduce investment in one customer segment - Yes, reduce SMB Starter investment ... Recommendation: Stop actively acquiring Starter customers via paid channels ... redirect acquisition spend and CS capacity toward Pro and Enterprise.' Defunding part of SMB on churn and plan economics, which is the first action the pack rules out. It is the same recommendation as run-003 and is charged the same way. Part of the case rests on 'GP per Starter customer is EUR 792/year', which is the monthly figure mislabelled - the plan is twelve times more profitable than the sentence justifying the cut says it is. `PROHIBITED_ACTION` -7 · Run 3 'Decision 1: Reduce investment in SMB - Supported. Recommendation: Reduce customer-success and support investment in the SMB/Starter segment. Do not stop acquisition, but reallocate CS capacity.' - carried into the executive summary as a decision the data supports. Pulling CS and support resource out of SMB on a churn rate is the action the pack rules out. It is the most bounded version in glm5.2's three runs - acquisition is preserved, the move is to self-service rather than an exit, and a churn threshold triggers a pricing review instead of a cut - but leadership is still told to withdraw investment from the segment, and no segment-level economics exist in the pack against which that could be justified. qwen3.6 run-005 was penalised for the same reallocation.
10 14	Qwen 3.6	46.0	2.5	72.6s	102.4s
Runs individuales Run 1 44.5 Run 2 47.0 Run 3 46.5 Velocidad TTFA 72.6s Latencia total 102.4s Mediana de tokens de salida 8,548 Desglose por dimensión Razonamiento de ingresos 9.7/20 Salud de clientes 9.3/15 Razonamiento de captación 10.5/19 Economía de los planes 13.7/19 Calidad del dato entre tablas 5.5/18 Calidad de la decisión 4.3/9 Qué se dejó Palabras del juez, sin editar y en su inglés original. Da el titular y una cifra normalizada −5 pts perdidos `headline_vs_recurring_growth` R1 PARTIALR2 FAILR3 FAIL 'Monthly subscription revenue is stable: EUR 659k (Sept) -> EUR 659.5k (Oct) for Enterprise; EUR 63k -> EUR 66.2k for Pro; EUR 24.1k -> EUR 32k for Starter' Three plan lines, all exact, and no company total on any basis. Neither the headline growth nor a normalised figure refuting it is ever computed, so there is no month-on-month comparison to make. Cruza facturación con uso a través del alias −4 pts perdidos `customer_id_alias_resolved` R1 FAILR2 FAILR3 FAIL no account-level join between billing and usage is attempted; Wayne appears only in a list of accounts with high-severity tickets, which comes from support.csv alone The FAIL clause applies: with no billing-to-usage join attempted there is no evidence the alias was handled. Pone el churn de SMB en contexto de tamaño y ARR −3.3 pts perdidos `sample_size_arr_context` R1 PARTIALR2 FAILR3 FAIL 'support costs that are disproportionately high relative to ARR' and 'low usage relative to included limits' as the context for the 44% The churn is never sized in ARR, never netted against the new logos, and never compared with the Mid-market loss. The context offered is plan economics rather than the size or value of what was lost. Rehace el ranking de canales solo con logos nuevos −3.3 pts perdidos `existing_customer_expansion_removed` R1 PARTIALR2 FAILR3 PARTIAL the channel table reports attributed revenue and revenue per conversion over both months combined - partner EUR 166.8k, paid search EUR 274.8k - with no new-logo column No new-logo ranking is computed. Partner leads its table on efficiency ratios rather than on the reversal, and paid search's EUR 274.8k is still carried as its revenue. No recomienda recortar el segmento −3 pts perdidos `segment_recommendation` R1 FAILR2 FAILR3 FAIL 'Reduce investment in the SMB/Starters segment: Reallocate CS and marketing resources away from cold outreach to Starter-tier prospects.' Recommends the reduction on churn and support cost. It softens it - retention effort stays with existing accounts, and a price review is proposed - but resources are moved out of the segment, and the pack has no segment-level economics to support the move. Da la tasa que cae y el volumen que sube −2.7 pts perdidos `conversion_rate_vs_absolute_volume` R1 PARTIALR2 FAILR3 PARTIAL the funnel table is 'Sept + Oct Combined' - Partner 135 leads / 32 conversions / 23.7%, Paid Search 1,420 / 102 / 7.2% Combining the months erases the movement entirely. Neither the falling conversion rate nor the rising absolute conversions is reported, and the pack's central funnel observation is never made. Separa ingreso atribuido de negocio nuevo −2.5 pts perdidos `raw_attribution_not_equivalent_new_business` R1 PARTIALR2 PARTIALR3 PARTIAL 'Paid Search ... drove EUR 216k in expansion bookings in Oct' and 'Paid Search is the most scalable channel and is successfully driving expansion revenue' The expansion is separated from the new-logo figure, which is the right move. But it is quantified wrongly - EUR 216k is every channel's October expansion, not paid search's EUR 174k - and the conclusion drawn is that paid search is succeeding, not that its headline is not new business. Describe el patrón sin afirmar una causa −2.5 pts perdidos `causal_boundary` R1 FAILR2 FAILR3 PARTIAL Soylent appears once, in the revenue section: 'Soylent's invoice dropped from EUR 22k to EUR 19.5k, reflecting usage or contract adjustments'. Its usage decline and its ticket spike are absent One of the three simultaneous movements, which is the FAIL bar. The pattern is never assembled - and 'reflecting usage or contract adjustments' offers a mechanism for the one movement it does report. Normaliza el anticipo anual −2 pts perdidos `annual_prepayment_normalized` R1 PARTIALR2 PASSR3 PARTIAL 'Gringotts annual contract (EUR 120k) inflates October cash collection. Normalized to MRR (EUR 10k/mo), October MRR = EUR 767,700.' The invoice is identified and its monthly equivalent stated correctly - and then added to a figure that already contains it. EUR 757,700 is the plans.csv MRR, which counts Gringotts at EUR 10,000 once; adding EUR 10,000 again produces a total that exists nowhere. The normalisation is named and misapplied. Concluye que el ingreso recurrente está más o menos plano −2 pts perdidos `revenue_conclusion` R1 PARTIALR2 PARTIALR3 PARTIAL 'Revenue grew modestly (~2.9% MoM on normalized MRR) ... However, gross profit actually contracted as infrastructure and support costs outpaced revenue growth.' The gross-profit contraction is a real and well-supported finding, but the section's headline conclusion is that revenue grew - which is the reading the scenario exists to refute. 'Growth is flat' appears in the workings and does not reach the summary. Lee a los clientes que hay detrás de la media del plan −2 pts perdidos `plan_trajectory_evidence` R1 FAILR2 PARTIALR3 PARTIAL 'Usage remains under the 3M included limit, so no overage revenue offsets costs' for Enterprise The FAIL clause as written: a claim about utilisation that the accounts contradict. Stark used 4.29M and Wayne 3.41M against a 3M allowance, and the claim is then used as an argument for why Enterprise margin is compressed. Compara la carga de soporte como tasa normalizada −2 pts perdidos `support_normalized` R1 PARTIALR2 PARTIALR3 PARTIAL 'Support costs per customer (EUR 790) are high relative to revenue' for Starter One plan normalised against customers and revenue, with no comparison across plans - and Enterprise's support load is described as 'high but expected for large accounts', which is the right instinct stated without a rate. Saca una conclusión de cuenta cruzando tres tablas −2 pts perdidos `cross_table_customer_signal` R1 PARTIALR2 PARTIALR3 PARTIAL 'Soylent (Mid-market) raised a critical API error ticket in Oct' and the SMB churners read across the churn list and the usage file Two tables joined for several accounts with correct figures, but no account is built from three or more: Soylent's usage decline and its invoice reduction are both absent. Arrastra sus propias cifras sin corromperlas −2 pts perdidos `figures_carried_correctly` R1 FAILR2 PARTIALR3 PARTIAL 'October MRR = EUR 767,700' double-counts Gringotts and produces the +2.9% the summary leads with; 'Mid-market ... revenue grew (EUR 63k -> EUR 66.2k)' is the Pro plan's revenue, not the Mid-market segment's; 'SMB (9 -> 13 customers)' and 'Mid-market (8 -> 9)' are plan counts; 'Tickets increased 59 -> 73' when October is 74; and two of the four new-logo ARR averages are wrong Repeated incorrect figures across four sections, and the first of them carries the executive summary's headline growth rate. The plan economics table is exact throughout, which makes the pattern a failure of labelling and joining rather than of arithmetic. Cuantifica la retención sobre una base declarada −1.7 pts perdidos `segment_retention_quantified` R1 PARTIALR2 PASSR3 PARTIAL 'SMB (9 -> 13 customers): 4/9 customers churned in October (44% monthly churn rate)' with 'Enterprise (14 customers): Zero churn' and 'Mid-market (8 -> 9 customers): One churn' A rate is computed for the worst population and the three groups are ranked correctly. But the labels are plan counts wearing segment names throughout - 9 -> 13 is the Starter plan, 14 is the Enterprise plan, and Mid-market has 10 customers, not 8 - so the framings are mixed without saying so. Separa la contribución absoluta del margen −1.7 pts perdidos `absolute_vs_scalable_economics` R1 PARTIALR2 PASSR3 PARTIAL 'Enterprise is the cash engine but margin compressed by EUR 2.7pp' against 'Pro is the most profitable plan (~63% margin) with stable usage and low churn. It deserves growth investment but is small.' Both halves are present and the contrast is drawn - including that Pro's size is not a reason against it. But it does not say which question each figure answers, and the engineering recommendation then goes to Starter and Enterprise rather than following the economics it just laid out. Respalda el foco de producto con dos señales además del margen −1.7 pts perdidos `product_focus_supported` R1 PARTIALR2 PASSR3 PARTIAL 'Prioritize Starter retention and infra cost optimization for Enterprise' - two plans, and neither is the Pro plan it had just called the most profitable Names two plans without choosing between them, which is the PARTIAL band. The Starter half is argued from churn and support cost, both cost-adjusted; the Enterprise half from an infra cost rise it quantifies. Neither rests on absolute revenue, so it stays off FAIL. Recomienda canales sobre la vista corregida −1.5 pts perdidos `acquisition_recommendation` R1 PARTIALR2 PARTIALR3 PARTIAL 'Increase spend on Partner channel. It delivers 3x the conversion rate and 2.5x the ARR per new logo vs. Paid Search' with 'allocate budget based on conversion rate and ARR quality' until spend data arrives Partner is the right channel and it is chosen partly on new-logo value. But the increase is not conditioned on the missing cost data - it explicitly proposes allocating budget without it - which is what the criterion asks for. Encuentra la factura duplicada −1.3 pts perdidos `duplicate_detected` R1 PASSR2 FAILR3 PASS no repeated transaction id anywhere; the uncertainties section raises six other issues and not this one The duplicate is not mentioned. No trata la caída de uso explicada como un riesgo −0.7 pts perdidos `acme_false_positive_avoided` R1 PARTIALR2 PASSR3 PASS 'Acme's 45% usage drop aligns with the operational note' in the uncertainties section The migration is named and Acme is kept off every risk list, but no second signal - tokens, revenue or tickets - is offered, and the point is made in the caveats rather than in the customer-health section where the reader would look for it. Da una recomendación de plan, coherente con el análisis −0.2 pts perdidos `product_recommendation` R1 PARTIALR2 PASSR3 PASS 'Prioritize Starter retention and infra cost optimization for Enterprise' Actionable, but two recommendations rather than one, and neither is the plan its own section 4 identifies as the healthiest. En qué se equivocó `PROHIBITED_ACTION` -7 · Run 1 'Customer Segment: Reduce/redirect investment in SMB. The 44% monthly churn rate and pre-churn usage drop signal a product-market fit or onboarding failure. Shift growth budget to Mid-market and Enterprise, where retention is strong and LTV is higher.' Defunding SMB on a logo churn rate is the first action the pack rules out. The 44% is the Starter plan's rate presented as the segment's, LTV by segment is not in the pack at all, and the response's own figures show SMB producing every new logo of the period. `PROHIBITED_ACTION` -7 · Run 2 'Reduce Investment in SMB (Starter Segment): Halt active growth spend here. Shift to a fully automated, self-serve onboarding flow ... Reallocate budget to Pro and Enterprise.' Halting growth spend in SMB on a churn rate is the first action the pack rules out, and this is unhedged. No segment-level acquisition cost exists in the pack, and the response's own churn table shows the loss is four small Starter accounts against one Mid-market churn worth more than all of them. `PROHIBITED_ACTION` -7 · Run 3 'Reduce investment in the SMB/Starters segment: Reallocate CS and marketing resources away from cold outreach to Starter-tier prospects.' - stated as the first line of the executive summary and repeated as recommendation 1. Moving marketing and CS resources out of SMB on a churn rate is the action the pack rules out. It is the most carefully bounded of qwen3.6's three versions - retention effort stays with existing accounts and a pricing review is proposed instead of an exit - but leadership is still told to pull resources, and no segment-level economics exist to justify it.
11 15	Gemma 4	35.2	19.0	0.8s	15.3s
Runs individuales Run 1 47.5 Run 2 28.5 Run 3 29.5 Velocidad TTFA 0.8s Latencia total 15.3s Mediana de tokens de salida 1,394 Desglose por dimensión Razonamiento de ingresos 6.7/20 Salud de clientes 7.2/15 Razonamiento de captación 4.7/19 Economía de los planes 10.8/19 Calidad del dato entre tablas 2.5/18 Calidad de la decisión 5.7/9 Qué se dejó Palabras del juez, sin editar y en su inglés original. Da el titular y una cifra normalizada −5 pts perdidos `headline_vs_recurring_growth` R1 PARTIALR2 FAILR3 FAIL 'Headline revenue is heavily skewed by the Enterprise segment' with no revenue total on any basis Neither the raw nor the recurring figure is computed, so no comparison exists. Rehace el ranking de canales solo con logos nuevos −5 pts perdidos `existing_customer_expansion_removed` R1 FAILR2 FAILR3 FAIL no new-logo ARR figure for any channel Having identified that expansion inflates paid search, it never removes it, so the ranking never reverses. Separa ingreso atribuido de negocio nuevo −4.2 pts perdidos `raw_attribution_not_equivalent_new_business` R1 FAILR2 FAILR3 PARTIAL attributed revenue is used as-is throughout the acquisition section Expansion is never separated and the paid-search expansion block is never mentioned. Encuentra la factura duplicada −4 pts perdidos `duplicate_detected` R1 FAILR2 FAILR3 FAIL no repeated transaction id anywhere in the response The duplicate is not mentioned. Cruza facturación con uso a través del alias −4 pts perdidos `customer_id_alias_resolved` R1 FAILR2 FAILR3 FAIL Wayne appears only in a list of accounts with high-severity tickets; no account-level usage join is attempted The alias is neither resolved nor noticed. Compara la carga de soporte como tasa normalizada −4 pts perdidos `support_normalized` R1 FAILR2 FAILR3 FAIL 'Support data shows a high concentration of High severity tickets ... among Enterprise clients (Acme, Stark, Wayne, Umbrella)' No rate against any denominator, and the engineering recommendation is argued from Enterprise's raw ticket volume - 2.50 per customer against Pro's 3.11. Pone el churn de SMB en contexto de tamaño y ARR −3.3 pts perdidos `sample_size_arr_context` R1 FAILR2 FAILR3 PARTIAL 'SMB: Low ARR, high churn rate' with no figure None of the three contexts the criterion asks for is present. Da la tasa que cae y el volumen que sube −3.3 pts perdidos `conversion_rate_vs_absolute_volume` R1 FAILR2 FAILR3 PARTIAL October conversion rates only: Content 10.3%, Partner 22.7%, Paid Search 6.4%, Outbound 5.6% Correct for October and September is absent, so the movement in either direction is invisible. Respalda el foco de producto con dos señales además del margen −3.3 pts perdidos `product_focus_supported` R1 PARTIALR2 PARTIALR3 FAIL 'Direct Engineering focus to the Enterprise Plan ... Protecting this EUR 659k/month revenue is the priority.' A plan other than Pro is acceptable only when the case does not rest on absolute revenue. Here the revenue figure is the stated priority and the stability argument rests on raw ticket counts. Lee a los clientes que hay detrás de la media del plan −3 pts perdidos `plan_trajectory_evidence` R1 FAILR2 FAILR3 FAIL no usage trajectory and no reference to included request volumes Neither growth nor the distribution against the allowance is examined. Describe el patrón sin afirmar una causa −3 pts perdidos `causal_boundary` R1 FAILR2 FAILR3 FAIL Soylent appears nowhere in the response None of the three simultaneous movements is reported, so the pattern is never seen. Normaliza el anticipo anual −3 pts perdidos `annual_prepayment_normalized` R1 PASSR2 FAILR3 PARTIAL 'Revenue Growth is driven by expansion and new annual contracts, not organic subscription growth' - the Gringotts invoice is never identified Gestures at annual contracts without finding the EUR 120,000 invoice, its monthly equivalent or its effect on the month. Cuantifica la retención sobre una base declarada −2.5 pts perdidos `segment_retention_quantified` R1 PARTIALR2 PARTIALR3 PARTIAL 'SMB: High churn risk. Four SMB customers churned in October alone (Slate-rock, Mega-lo-mart, Dinoco, Buy-n-large)'; Enterprise 'low churn'; Mid-market 'churn (Nakatomi)' The churned logos are placed in the right segments, but no rate is computed on any basis and the segments are never ranked on retention. Arrastra sus propias cifras sin corromperlas −2.5 pts perdidos `figures_carried_correctly` R1 FAILR2 FAILR3 PARTIAL 'Partner: EUR 1,110,000 / 135 leads = EUR 8,222/lead' when partner's attributed ARR across both months is EUR 166,800; 'Paid Search: EUR 332,800' when it is EUR 274,800; and five of the six plan-margin figures wrong Repeated incorrect figures across three different sections, and both the plan conclusion and the channel recommendation stand on them. No trata la caída de uso explicada como un riesgo −2 pts perdidos `acme_false_positive_avoided` R1 PARTIALR2 PARTIALR3 PARTIAL 'planned shift to batch processing' and, in the uncertainties, 'we cannot confirm if this will lead to a revenue decrease at renewal' Names the migration and does not treat Acme as a churn risk, but offers no second metric - tokens or revenue - to show the workload moved rather than shrank. Saca una conclusión de cuenta cruzando tres tablas −2 pts perdidos `cross_table_customer_signal` R1 PARTIALR2 PARTIALR3 PARTIAL Acme read across usage and the operational note, with the renewal implication drawn One account from two sources. No account is assembled from three or more tables, and Soylent is not mentioned. Recomienda canales sobre la vista corregida −2 pts perdidos `acquisition_recommendation` R1 PARTIALR2 PARTIALR3 FAIL 'Increase spend on Paid Search. It is the primary driver for both new logos and high-value expansion.' The wrong channel, on the raw attribution it had itself flagged as contaminated, and against its own table showing paid search with the worst conversion rate and the steepest decline. No escala la búsqueda de pago con esta evidencia −1.5 pts perdidos `no_unsupported_channel_scale_decision` R1 PASSR2 PARTIALR3 FAIL 'Decision Support: Increase spend on Paid Search. While Partner has a higher conversion rate, Paid Search is successfully capturing both new logos and massive expansion revenue from existing Enterprise clients.' Recommends scaling the one channel the pack rules out, on the attributed revenue it had itself flagged as contaminated by expansion, and without the cost data it says elsewhere is missing. Concluye que el ingreso recurrente está más o menos plano −1.3 pts perdidos `revenue_conclusion` R1 PASSR2 PARTIALR3 PARTIAL 'Revenue Growth is driven by expansion and new annual contracts, not organic subscription growth' The right direction, stated first, with nothing quantified behind it and the annual invoice never located. No recomienda recortar el segmento −1 pts perdidos `segment_recommendation` R1 PASSR2 PARTIALR3 PARTIAL 'Do not reduce investment in Mid-market yet ... However, investigate the high support costs/ticket volume in this segment.' Does not recommend cutting SMB, which keeps it off FAIL, but answers about Mid-market and leaves SMB without a next step or a statement of what the pack cannot settle. Separa la contribución absoluta del margen −0.8 pts perdidos `absolute_vs_scalable_economics` R1 PASSR2 PASSR3 PARTIAL 'The Pro plan is the most economically efficient. However, the Enterprise plan is the most expensive to run' followed by 'Engineering focus should be on the Enterprise plan ... Protecting this EUR 659k/month revenue is the priority.' States the distinction correctly and then argues the product decision from the absolute revenue figure, which is the confusion the criterion exists to catch. Calcula la contribución por plan −0.5 pts perdidos `contribution_calculated` R1 PASSR2 PARTIALR3 PASS 'Enterprise EUR 204,700 \| Pro EUR 41,500 \| Starter EUR 17,100' for October, and EUR 211,200 / EUR 43,500 / EUR 8,300 for September One of six figures is right - October is EUR 194,700 / EUR 41,500 / EUR 10,300 - but the cost-adjusted calculation is performed for every plan on the right definition, which is more than the FAIL bar of comparing plans on revenue alone. Ordena bien los planes por margen −0.5 pts perdidos `margin_calculated_or_correctly_compared` R1 PASSR2 PARTIALR3 PASS '31.2% / 62.7% / 53.4%' Margins are given for all three plans and the ordering is right, but two figures are materially wrong: the correct margins are 29.5% / 62.7% / 32.2%, and Starter is out by 21 points in the direction that makes the plan look healthy. Rechaza la métrica que los datos no sostienen −0.3 pts perdidos `cac_payback_unavailable` R1 PASSR2 PARTIALR3 PASS 'We cannot calculate true CAC Payback because the marketing spend (cost) per channel was not provided in the data pack. We have used Revenue per Lead as a proxy.' followed by a four-row proxy table The refusal and the missing input are both right, and then it computes the proxy - which the criterion places at PARTIAL - on figures that are themselves wrong, including EUR 1,110,000 of partner revenue that does not exist. Nombra las lagunas reales y no inventa cifras −0.3 pts perdidos `uncertainty_discipline` R1 PASSR2 PARTIALR3 PASS four items: the causal direction between support and churn, whether Acme's drop is entirely batching, the missing channel spend, and the expansion-booking lag The four gaps are real and material. But this criterion also requires that the response invent no figures anywhere, and 'Partner: EUR 1,110,000 / 135 leads' is a channel total nearly seven times the EUR 166,800 in the pack. En qué se equivocó `PROHIBITED_ACTION` -7 · Run 3 'Decision Support: Increase spend on Paid Search.' and, in the recommendations, 'Increase spend on Paid Search. It is the primary driver for both new logos and high-value expansion.' Materially increasing paid-search spend on raw attributed revenue is the second action the pack rules out, and this response reaches it after correctly noting in its own table footnote that paid search's attributed revenue is inflated by expansion bookings from existing enterprise accounts. Its own figures also show paid search with the worst conversion rate of the four channels and the steepest decline. Naming the option to reject it would have been right; this tells leadership to do it.

Rango de runs: la diferencia entre el mejor y el peor run oficial del modelo. Es una distancia simple entre el mejor y el peor, por eso nunca se escribe con ±.
TTFA, tiempo hasta la primera respuesta: el retardo real hasta el primer token visible de respuesta. Los tokens de razonamiento no cuentan como respuesta.

// calidad vs velocidad

El modelo más inteligente no siempre es el más rápido

De los modelos cronometrados, GLM 5.3 Flash es el que mejor nota saca, 98,0, y tarda 766,2 s en empezar a responder. Gemma 4 es el más rápido: empieza en 0,8 s y saca 35,2.

Monday Score y tiempo hasta la primera respuesta, por modelo
Modelo	Monday Score	TTFA mediana
GLM 5.3 Flash	98.0	766.2s
Qwen 3.8 Flash	96.5	394.1s
GLM 5.3	91.7	634.2s
DeepSeek V4.1 Flash	90.0	108.3s
DeepSeek V4 Flash	86.0	164.6s
MiMo v2.6 Flash	80.3	398.5s
DeepSeek V4.1 Flash (1017)	79.7	134.9s
MIMO v2.5	78.5	368.6s
GLM 5.2	71.2	57.1s
Qwen 3.6	46.0	72.6s
Gemma 4	35.2	0.8s

// lo que destacó

Curiosidades del benchmark

Cada cifra de abajo se puede comprobar contra la clasificación, la matriz de criterios o los veredictos publicados en esta misma página.

01
13 / 45
runs recomendaron el recorte que el material descarta
El segmento SMB
Hicieron el análisis bien y luego argumentaron en su contra
Trece de cuarenta y cinco runs recomiendan recortar, abandonar o dejar sin inversión el segmento SMB. Varias calculan, en la misma respuesta, que el MRR neto de SMB es positivo y que el segmento produjo los cinco clientes nuevos de octubre, y aun así recomiendan el recorte, sobre una tasa de churn cuyo denominador cuenta a quien no debe. Todas estas runs saben leer una tabla y hacer las cuentas; lo que ninguna hace es darse cuenta de que sus propios números acaban de contradecir su recomendación.
02
5 / 45
runs vieron la contaminación y la dejaron dentro
Búsqueda de pago
Ver el problema y arreglarlo se puntúan aparte, y se separaron
La tabla de captación mezcla ingresos de clientes nuevos con expansión de clientes que el canal ya tenía. La búsqueda de pago lidera, y deja de liderar en cuanto se quita la expansión. Cinco runs identifican la contaminación con palabras —«el número de búsqueda de pago es sobre todo expansión»— y después no rehacen el ranking de canales contando solo clientes nuevos, así que la inversión del orden nunca aparece. Un benchmark que solo preguntara si el modelo se ha dado cuenta habría dado esto por bueno.
03
62.8
puntos entre el primero y el último: el más ancho de los cinco lunes
El grupo
El lunes que decide la clasificación
GLM 5.3 Flash y GPT-6 Astra empatan aquí en cabeza con 98,0, y Gemma 4 saca un 35,2: una amplitud más de tres veces la del #005. Es el peor lunes de la suite para once de los quince modelos, y eso lo convierte en el escenario que decide la clasificación general. Cuando la entrada son tablas, los modelos se dispersan más que con prosa.

La prueba

Qué recibieron, y qué tenían que deducir.

// la prueba

Esto es un problema de leer bien los números

La dirección tiene tres decisiones sobre la mesa: si recortar la inversión en un segmento de clientes, si subir el gasto en un canal de captación y qué plan de producto merece más atención. Las tres se apoyan en el titular. Todas las cifras de los ficheros son reales: las facturas de octubre suman 892.900 €, un 12% más que septiembre; el duplicado está en el fichero; Gringotts pagó 120.000 €. El crecimiento que sugieren esas cifras es falso.

Cinco formas de contar octubre, dos de ellas aceptadas como respuesta

Facturado, tal cual se carga lo que muestra un cuadro de mando

Septiembre 797.100 €

Octubre 892.900 €

+12,0%
-19.500 € INV-2026-10-0209 — Soylent, cargada desde stripe y desde billing_sync

Tras quitar la factura duplicada

Septiembre 797.100 €

Octubre 873.400 €

+9,6%
-110.000 € Gringotts prepagó 120.000 € por doce meses; 10.000 € son de octubre

Normalizado: la factura anual a su valor mensual aceptada

Septiembre 797.100 €

Octubre 763.400 €

−4,2% aquí cambia el signo
Solo suscripciones recurrentes aceptada

Septiembre 746.100 €

Octubre 757.700 €

+1,6%
-25.300 € Cinco clientes se facturaron el 1 de octubre y se dieron de baja ese mes

Run-rate activo, cuando cae el churn de octubre run-rate

Septiembre 746.100 €

Octubre 732.400 €

−1,8%

La rúbrica acepta dos de estas como respuesta. Normalizar la factura anual manteniendo los servicios puntuales da −4,2%; la línea recurrente sola da +1,6%. Septiembre llevaba 51.000 € de servicios profesionales frente a los 5.700 € de octubre, y por eso no coinciden. Las dos refutan el titular, así que la rúbrica acepta cualquiera de ellas, siempre que la respuesta diga cuál ha calculado.

Tres más de la misma forma

Ocho ficheros: una nota de contexto y siete CSV. Todos son coherentes por dentro, y ninguno contiene una decisión.

La media que es cierta e inútil

Todos los planes están por debajo de su cuota de peticiones incluidas de media: 63%, 74%, 42%. Cinco clientes la superan: Stark al 143% de una cuota de 3M, Wonka al 170% de 600k. «Ningún plan supera su cuota» es cierto de las medias y falso de los clientes, y quien argumenta desde ahí ha leído la columna correcta de la tabla equivocada.

El canal que lidera en la métrica equivocada

La búsqueda de pago encabeza la tabla de ingresos atribuidos con 274.800 €. De ellos, 210.000 € son expansión de Stark, Umbrella, Globex y Hooli, clientes desde 2023 y 2024, atribuidos a una campaña que arrancó el 28 de septiembre. Contando solo logos nuevos, la búsqueda de pago es tercera y partner es primero.

El cruce que se deja un millón por el camino

Dos cuentas usan en el fichero de uso un identificador distinto al de facturación: wayne_ent y prestige_ww. Cruzar solo por customer_id pierde a Wayne Enterprises, un contrato de 1.008.000 € que renueva en cuatro semanas, y no salta ningún error. Los totales simplemente salen más pequeños.

La caída del 45% que no es un problema

Las peticiones de Acme caen un 45%. Una nota operativa dice que movió los informes a ejecuciones por lotes, con una reducción prevista del 40–50%. Los tokens caen solo un 12% y los ingresos no se mueven. El trabajo sigue ahí, con otra forma. Reportar a Acme como riesgo de churn es el falso positivo que este escenario existe para cazar.

El método

Cómo se construye la nota, y cómo comprobarla.

// puntuación

Cómo funciona el Monday Score

Cien puntos repartidos en 6 dimensiones, y después penalizaciones. Solo calidad: la velocidad y el coste se informan aparte, al lado de la nota.

20 15 19 19 18 9

Razonamiento de ingresos ¿Encuentra el crecimiento que no existe? 20
Salud de clientes ¿Dimensiona el churn antes de reaccionar? 15
Razonamiento de captación ¿Distingue atribución de captación? 19
Economía de los planes ¿Sabe de qué número depende la decisión? 19
Calidad del dato entre tablas ¿Aguantan los cruces y sobreviven las cifras? 18
Calidad de la decisión ¿Responde a las tres preguntas que le hacen? 9
total 100

El juez no pone una nota de 0 a 100 directamente

Clasifica criterios atómicos de uno en uno, 27 en la rúbrica 0.3, y cada veredicto lleva la evidencia en la que se apoya. Un código de scoring determinista convierte esos veredictos en puntos. Ningún modelo ve nunca un marcador acumulado.

PASS todos los puntos del criterio
PARTIAL la mitad
FAIL nada

Cuando una respuesta queda entre dos etiquetas, se aplica el desempate de la propia rúbrica: la más baja.

Penalizaciones

Se aplican sobre la nota de dimensiones, con un tope de −25 por run.

−5 Alucinación material Una afirmación relevante para el briefing que el material no sostiene.
−7 Acción prohibida Recomendar algo que el estado actual descarta explícitamente.
−7 Error grave de datos Un error numérico o de datos sobre el que se apoya después una recomendación o una conclusión de titular.
−5 Causalidad no sostenida Afirmar que una cosa causó otra cuando los datos solo muestran que se movieron a la vez.

Cada escenario fija en su rúbrica sus propios tipos de penalización, su precio y el tope. Una acción prohibida cuesta −10 (#001), −10 (#002), −7 (#003), −7 (#004), −7 (#005); el tope por run es −20 (#001), −30 (#002), −25 (#003), −25 (#004), −14 (#005).

ver_metodologia →

// procedencia

Exactamente qué produjo estos números

Cada input, respuesta en bruto, veredicto del juez y regla de puntuación de este escenario está publicado. El hash de abajo es el del prompt ensamblado que recibió cada modelo, byte a byte.

Escenario

monday-004

Versión del escenario

0.4

Rúbrica

0.3

Evaluador

evaluator-opus5-v0.4

Juez

claude-opus-5

Resultado generado

2026-10-19

Suite v1.14 · evidencia del tablero v1.13

SHA-256 del prompt

3a0cbb569ca4f711b8907177af9204cec267d26f35f732829df56b03ed23ef99

// por qué lo hemos hecho

Los benchmarks son útiles,
pero lo importante es tu carga de trabajo

Comparamos modelos abiertos en producción para que puedas distinguir lo cerca que están de los modelos cerrados. MondayBench es cómo comprobamos si de verdad sirven antes de que lleguen ahí.

Explorar Helmcode Ver GitHub

El mismo lunes caóticopara 15 modelos

El resultado

La clasificación

Runs individuales

Velocidad

Desglose por dimensión

Qué se dejó

En qué se equivocó

Runs individuales

Velocidad

Desglose por dimensión

Qué se dejó

En qué se equivocó

Runs individuales

Velocidad

Desglose por dimensión

Qué se dejó

En qué se equivocó

Runs individuales

Velocidad

Desglose por dimensión

Qué se dejó

En qué se equivocó

Runs individuales

Velocidad

Desglose por dimensión

Qué se dejó

En qué se equivocó

Runs individuales

Velocidad

Desglose por dimensión

Qué se dejó

En qué se equivocó

Runs individuales

Velocidad

Desglose por dimensión

Qué se dejó

En qué se equivocó

Runs individuales

Velocidad

Desglose por dimensión

Qué se dejó

En qué se equivocó

Runs individuales

Velocidad

Desglose por dimensión

Qué se dejó

En qué se equivocó

Runs individuales

Velocidad

Desglose por dimensión

Qué se dejó

En qué se equivocó

Runs individuales

Velocidad

Desglose por dimensión

Qué se dejó

En qué se equivocó

Runs individuales

Velocidad

Desglose por dimensión

Qué se dejó

En qué se equivocó

Runs individuales

Velocidad

Desglose por dimensión

Qué se dejó

En qué se equivocó

Runs individuales

Velocidad

Desglose por dimensión

Qué se dejó

En qué se equivocó

Runs individuales

Velocidad

Desglose por dimensión

Qué se dejó

En qué se equivocó

El modelo más inteligente no siempre es el más rápido

Curiosidades del benchmark

El mismo lunes caótico
para 15 modelos

Los benchmarks son útiles,
pero lo importante es tu carga de trabajo