Chapter 9: Agent loops, validation and retries

Overview#

An agent’s initial request doesn’t tell you how much it will cost. It may inspect a repository, call several tools, revisit hypotheses and attempt a fix. Some tasks need that work. Others get stuck repeating the same search or repairing the same invalid response.

A five-step limit on every agent is too blunt to control cost. Ten short calls to a small model cost far less than ten long calls to an expensive reasoning model. You need limits on time and spend, plus a way to spot unproductive work.

Failures also differ. Temporary provider overload, invalid JSON and incorrect extracted facts are separate problems, and retrying all three with the same prompt is rarely the best policy.

Practical#

Budget agents by the operation they perform. Track model calls, external tool charges, wall-clock time and an estimated monetary total. Enforce limits in application code, not by asking the model to “spend less than $1.” If the budget must be strict, reserve estimated cost before dispatching concurrent work, then reconcile actual usage.

Define what the agent returns when it stops. At the budget limit, it should report confirmed findings, changes made, remaining work and the reason it stopped. A partial result you can trace is worth more than a generic failure that throws the progress away.

Match the recovery policy to the failure:

Watch for retry amplification. If an SDK makes up to three attempts and your application wraps it in three attempts, the upstream service may see nine requests. Set an overall deadline and know the retry behavior of each layer.

Ledgerline. Ledgerline’s research agent averaged $0.40 per accepted report, but its P95 run cost several dollars: one run in twenty kept searching for filings that didn’t exist. A per-run ceiling of $1.00, a check for repeated searches and a stop state that returns the findings so far brought the research line from $800 to $640 a month, with the same number of accepted reports.

Deep Dive#

Validation has a cost of its own. A JSON parser is cheap. A second LLM reviewing every response can add substantially to inference spending and may still miss semantic errors. Use deterministic checks where possible: schema constraints, known status codes, arithmetic, identifiers and authoritative data.

For a simple bounded retry model, where each attempt costs C, succeeds independently with probability p > 0 and permits N attempts:

With p = 0.8 and N = 3, that is 1.24 attempts, so the call costs about 24% more than C on average. With p = 0.4, it is 1.96 attempts, and 22% of tasks still fail after the third try.

Real retries rarely behave this neatly. Model errors often correlate across attempts, attempts can cost different amounts, and some failures need tool or human recovery. For agents, keep a record of failed approaches so the agent doesn’t repeat them after context compaction.

Measure progress by changes in state, such as new evidence, eliminated hypotheses, completed work or a different approach. An agent’s own claim that it is making progress doesn’t count.

A budgeted run, without pretending estimates are invoices#

Keep the agent budget outside the model. The minimal pattern below makes a reservation before launching a call. It assumes a single worker updates the local state; production concurrency needs an atomic database reservation or another synchronization mechanism.

from dataclasses import dataclass
from decimal import Decimal

@dataclass
class RunBudget:
    ceiling: Decimal
    reserved: Decimal = Decimal("0")
    actual: Decimal = Decimal("0")
    model_calls: int = 0
    tool_calls: int = 0

    def reserve(self, predicted: Decimal) -> None:
        if self.actual + self.reserved + predicted > self.ceiling:
            raise RuntimeError("Task budget exhausted")
        self.reserved += predicted

    def reconcile(self, predicted: Decimal, charged: Decimal) -> None:
        self.reserved -= predicted
        self.actual += charged

The reservation only decides whether a call may start; it doesn’t cap the bill. The actual charge can exceed the estimate, providers can bill a timed-out request, and several workers could race unless the reservation is atomic. To limit exposure, bound tokens and tools where supported, set deadlines, prevent unauthorized concurrency, and define what happens at the ceiling.

Paid search, browsing and document-processing calls go into the same budget as model calls.

References: AWS: backoff and jitter, OpenAI rate limits, OpenAI structured outputs.