Chapter 3: Choose the model the task needs

Overview#

Teams often pick a capable model for the first feature and use it everywhere afterward. Months later, that model is classifying tickets, summarizing conversations, extracting fields and investigating production incidents.

The setup is easy to understand, but some operations end up paying for capabilities they don’t need.

A five-category ticket classifier and an agent debugging an intermittent payment failure are different problems, and they rarely need the same model and configuration. For each operation, price the whole path to an accepted result, including validation, retries and escalation, before you pick a model by its token price.

Practical#

Group work by what it requires: constrained classification, extraction, summarization, grounded Q&A, multi-document reasoning and tool-using agents. For each group, record the acceptance criteria and the non-negotiable constraints: latency, context size, output format, language, availability and data handling.

Then shortlist models that can meet those requirements, including open-weight candidates where appropriate. An overall benchmark ranking doesn’t tell you whether a model suits the task.

To find out, build a small evaluation set from real cases of that operation, including tasks that were accepted and tasks that were rejected. Run each candidate against it before switching, and again after any prompt or model change. Those runs cost tokens and review time, so count them as part of the decision. Look at results by case type and failure mode as well as the overall score, because a model can match the average while failing the cases that matter most.

A hypothetical comparison of two approaches to the same extraction operation:

Both columns include validation because either model can return a wrong answer. The comparison also assumes equal acceptance rates; if they differ, compare cost per accepted task instead. Whether the cheaper column holds up depends on how reliably the check catches failures, which Routing, cascades and reasoning budgets covers.

Before adding a different model, check whether ordinary code can do the job. Formatting known fields, comparing timestamps and arithmetic usually don’t need generative inference.

Ledgerline. Ledgerline built an evaluation set of 300 real invoices, 200 accepted and 100 previously rejected, and ran two candidates against it. Each run of the set costs about $9. A smaller model with 15% escalation to the large one matched the large model’s acceptance on the set and brought extraction to $0.018 per accepted invoice: 40,000 × $0.018 = $720 a month, down from $1,200.

Deep Dive#

Compare billable token usage as well as published unit prices. Two models can tokenize the same input differently, need different levels of prompt detail, produce outputs of different length and consume different reasoning budgets.

A larger context window doesn’t guarantee better use of a long document; retrieval quality and prompt structure may matter more than the headline limit. Every additional model also adds maintenance: provider configurations, prompt variants, fallback behavior and monitoring.

Fine-tuning or distillation is another lever. On a narrow, stable task, such as one extraction schema that rarely changes, a smaller model trained on your accepted outputs or on a larger model’s answers can replace a large general one. It brings its own costs: preparing training data, training runs, evaluation, hosting and retraining when inputs or requirements change. The licenses of the models involved must also allow it, which What open-weight models actually change covers. Spread those costs over the volume you expect before the next retraining and add them to the comparison.

Keep the number of models small, and add one for a distinct workload only when the benefit justifies that maintenance.