Chapter 7: Agent memory, response caching and duplicate work

Overview#

A coding agent reads files, checks logs and tries approaches that fail. An hour in, its transcript may hold thousands of lines of output. The next step rarely needs all of them, only the objective, the constraints, confirmed findings, failed approaches and where the evidence lives. That smaller set is the agent’s working state.

Context compaction, response caching and idempotency each avoid a different kind of repeated work, so treat them as separate problems.

Practical#

Maintain three layers: recent conversation for continuity, durable working state for decisions and constraints, and evidence stored elsewhere with retrievable references. A compact record might look like this:

{
  "objective": "Fix the webhook retry bug",
  "constraints": ["No duplicate charges", "Rollback required"],
  "findings": ["Worker may time out after downstream success"],
  "failed_approaches": ["Blind retry duplicates the action"],
  "evidence": ["src/webhooks.py", "incident-1042"],
  "next_step": "Check downstream idempotency support"
}

Keep exact user constraints and authoritative business state outside lossy prose summaries. Compaction costs tokens and may break prompt-prefix reuse, so compact when the expected reduction in active context justifies it, not automatically after every turn.

For completed operations that can safely produce the same answer, use exact response caching keyed by the relevant inputs and versions. A document extraction key might include content hash, schema version, preprocessing version, model configuration and tenant scope. Enforce authorization separately, since a cache key doesn’t control access.

When two identical requests arrive at the same time, a finished-result cache alone won’t help, because both may miss and start inference. Use in-flight deduplication or a shared operation record so callers wait on the same work.

Measure duplicate work before building a cache. Log a request hash on every model call, computed from the inputs a cache key would use. Each day, report per operation the share of calls that repeat an earlier hash and what those calls cost. Count identical requests that overlapped in time separately, since only in-flight deduplication removes those. That report tells you whether exact caching is worth building.

Illustrative: an extraction operation makes 40,000 calls a day, and 6,000 repeat a hash seen earlier that day: 6,000 / 40,000 = 15%. At $0.004 per call, the repeats cost 6,000 × $0.004 = $24 a day, or $720 over 30 days.

Ledgerline. About 10% of invoices reaching Ledgerline are the same file uploaded twice, by the supplier and by the customer. A content hash plus the schema and model version now keys an exact cache, scoped to the customer account, so 4,000 of the 40,000 monthly invoices skip inference. Extraction goes from $720 to $648.

Deep Dive#

Response caching asks, “Do we have a still-valid result for these inputs?” Idempotency asks, “Has this particular intended operation already been submitted?” An agent that retries sending an email after losing a tool response must not send two emails just because the text is identical. Preventing that takes a durable operation identity.

A distributed lock doesn’t prevent duplicate charges either. If a worker crashes or times out after sending a request, the provider may still run it, and the retry runs it again. You can guarantee that only one result is committed and still pay for two upstream calls.

Semantic caching is riskier than exact caching. “Can I cancel?” and “Can I cancel and get a refund?” are similar questions that need different answers. Similarity alone doesn’t establish equivalence, freshness or authorization. Start with exact reuse and deduplicated execution; add semantic reuse only when you can define safe boundaries.

How transcript growth adds up#

An agent that resends its whole transcript pays for every earlier step again on each new one. If each step adds t tokens, step k sends about k × t input tokens, and a run of n steps sends t × (1 + 2 + … + n) = t × n(n + 1) / 2. Total input grows roughly with the square of the number of steps. An agent that sends a capped working state plus the latest step sends about the same amount each time, so its total grows linearly.

Illustrative: each step adds 2,000 tokens. The compact agent sends a 4,000-token working state plus the latest 2,000-token step, 6,000 tokens per step.

Doubling the run from 50 to 100 steps multiplies full-transcript input by about four and compact input by two. The compact column leaves out the tokens spent updating the working state. Prompt caching lowers the price of the repeated prefix (Tokens, context and prompt caching), but the token count grows the same way.

One operation, one in-flight result#

A cache that checks get(key) and runs inference on a miss lets two simultaneous requests launch two calls. Coordination has to happen before the expensive work starts.

A durable task record can have these fields:

(tenant_id, idempotency_key)  UNIQUE
request_hash                   required
status                         pending | running | completed | failed
result_reference               nullable
lease_owner, lease_expires_at  nullable

On arrival, atomically create or retrieve the record. If the key exists with a different request_hash, reject the mismatch. If it is completed, return its result; if it is running, return the task identifier or wait for completion. A lease allows controlled recovery when a worker disappears. Publish the final result only if the worker still owns the lease; a database transaction or fencing token can protect that update.

This protects the application’s state, but it is not an exactly-once guarantee for the inference provider. If the worker loses its lease after dispatching a remote request, a recovery worker may dispatch another one, so record both attempts if they incur charges. For tools that create a payment, send an email or change a remote record, use the downstream system’s idempotency contract wherever one exists.

Keep this separate from the finished-result cache. A document hash plus extraction-schema version may justify reusing a result across separate legitimate operations, while an idempotency key identifies retries of the same intended operation. Mixing the two causes mistakes when a user deliberately asks for fresh processing.

References: AWS: idempotent APIs, Redis: distributed locks, OpenAI conversation state.