Chapter 4: Routing, cascades and reasoning budgets

Overview#

Routing and cascading are related techniques. Routing chooses a model for a request. A cascade starts with one model and escalates after a detectable failure or insufficient result.

If your application has separate endpoints for classification and complex analysis, it probably already knows which model each one needs. A deterministic lookup is simpler than asking an LLM to route another LLM.

Dynamic routing makes sense when one endpoint receives requests with very different requirements. Even then, the two kinds of mistake cost different amounts: an easy task sent to a costly model wastes money, while a difficult task sent to an unsuitable one may fail the whole operation.

Practical#

Start with static policies:

MODEL_BY_OPERATION = {
    "classification": "small-model",
    "extraction": "extraction-model",
    "incident_analysis": "reasoning-model",
}

def choose_model(operation: str) -> str:
    return MODEL_BY_OPERATION.get(operation, "default-model")

When a cascade is justified, its simplified expected cost is:

For example, a $0.005 initial call, 20% escalation to a $0.020 model and $0.001 validation produces an expected $0.010 before other recovery work. That figure depends on your escalation signal and acceptance requirements.

Some reasoning models expose effort or thinking controls, which can cut unnecessary internal generation on constrained tasks without switching models. Supported values, defaults and billing vary by model and provider; don’t assume a generic none, low or high setting works everywhere.

Set reasoning effort per operation, next to the model choice. A basic classifier may need little extended reasoning, while a difficult code investigation may justify more. Don’t use a low output-token limit as a stand-in for low effort: it can truncate the final answer after internal reasoning has already consumed the budget.

Ledgerline. Six in ten Ledgerline support questions ask where an invoice is in the approval flow. The application already knows that from its own API, so those 18,000 questions now go through a lookup and a small model with low reasoning effort at $0.010 each. The other 12,000 keep the current setup at $0.020. Support falls from $600 to 18,000 × $0.010 + 12,000 × $0.020 = $420 a month, with no router model.

Deep Dive#

Dynamic routing adds classification cost and latency. If the router costs $0.001 and avoids $0.003 of downstream inference on average, the gross difference is $0.002 per request, before wrong routes and maintenance. At small scale, the architecture may not be worth it.

Cascades depend on a useful escalation signal. Schema checks and business-rule violations are cheap to detect; semantic correctness is harder. A model that reports high confidence can still be wrong, and asking another model to judge every answer may erase part of the saving.

Reasoning-effort controls affect the whole workflow. With less reasoning, an agent may make more tool calls or repeat work. On a metered API, fewer billable reasoning tokens can lower a call’s charge; on dedicated capacity, the immediate benefit is more likely freed throughput.