Chapter 6: Tokens, context and prompt caching

Overview#

Prompts grow for understandable reasons: a new example fixes a corner case, another instruction improves consistency, and at some point the application starts including the entire conversation to preserve continuity. Eventually every request carries material nobody can clearly justify.

Start by asking why each piece of information is there. Some of it defines the task, some is evidence, and some is repeated because the application never learned to leave it out. Cutting irrelevant context can lower cost and make the model’s job clearer, as long as the evidence the answer depends on stays in.

Output counts too. If the product needs a category and priority, an essay explaining the choice may be waste. Specify the exact output format, since “be concise” is too vague.

Practical#

Inspect the actual request sent to the model: system instructions, examples, tool definitions, retrieved passages and conversation history, not just the visible user message.

For a support assistant, the current question, applicable policy and relevant account facts may be enough, without years of history. For extraction, prefer an explicit schema to free-form explanations the product discards. Keep exceptions, original identifiers and any evidence needed for correctness.

A cost example with hypothetical prices of $1 per million input tokens and $4 per million output tokens:

That’s a modeled 45.7% reduction for the call, assuming both results are equally acceptable and all input uses uncached pricing.

Prompt caching complicates this. When many requests share a long stable prefix, a provider may process and charge for that portion differently, and a shorter prompt with dynamic data near the beginning can lose that benefit. Put reusable instructions and reference material before request-specific content when that matches the provider’s caching behavior.

Images, audio and documents#

Non-text input follows its own billing rules. Images are usually converted to tokens based on resolution, often by splitting them into fixed-size tiles, so a larger image costs more even when the extra detail doesn’t help the task. Audio is typically counted by duration or converted to tokens. PDFs and other documents are often processed page by page, sometimes as both extracted text and a rendered image of each page.

A few habits keep these inputs in check:

• Send the resolution the task needs. Downscale screenshots and photos, and crop to the region that matters, such as one table or one form field.
• Avoid sending the same image or page again on every turn. Conversation history carries attachments forward, so store the first result (a description, extracted fields or a transcript) and reference it in later turns.
• When the task only needs the text, extract it with a cheap OCR tool or a document parser and send that text to the model.
• Check how the provider counts each modality before estimating. Tokens per image or per page vary by provider and model, and an estimate based on text alone can be far off.

Ledgerline. Every Ledgerline support request carried a 6,000-token block of policy text, with the customer’s name and a timestamp at the top. Moving the request-specific fields to the end made the policy block a stable prefix that the provider caches, and the answer format went from free prose to a short structured reply. Support drops from $420 to $330 a month.

Deep Dive#

Three mechanisms are easy to confuse:

With prompt or prefix caching, the model generally still has to generate a new answer. It helps most when long shared inputs recur often. There is no universal discount. Cache writes, reads, TTLs, minimum sizes and eligibility vary by provider and model.

For illustration: 10,000 cached input tokens at $0.10/M plus 2,000 uncached at $1/M cost $0.003, while a rewritten 8,000-token uncached prompt costs $0.008. In this scenario the shorter prompt costs more.

Serving-engine prefix caching works one layer down. Every request builds a KV cache, the attention state for the tokens processed so far, and every decode step reads it. Prefix caching, as in vLLM, keeps the KV cache of a shared prefix and reuses it for later requests that start with the same tokens. That saves prefill work for the prefix, but the model still generates every output token. On dedicated infrastructure, the benefit may be higher throughput rather than a smaller monthly invoice.

A stable prompt boundary in practice#

The simplified Python pattern below targets an API that supports ordinary system and user messages. Shared material stays stable and request-specific content goes last. The code does not guarantee a cache hit, which depends on the serving provider, the model, minimum eligible length and prior requests.

SYSTEM = (
    "You answer questions about our public API. "
    "Use only the supplied reference material. "
    "If the source does not answer the question, say what is missing."
)

# Versioned and stable across requests until the documentation changes.
DOCS_VERSION = "api-docs-v17"
REFERENCE = load_documentation(DOCS_VERSION)

def messages_for(question: str) -> list[dict[str, str]]:
    return [
        {"role": "system", "content": SYSTEM},
        {"role": "user", "content": f"Reference ({DOCS_VERSION}):\n{REFERENCE}"},
        {"role": "user", "content": f"Question:\n{question}"},
    ]

In a real application, load_documentation stands for your own versioned document loader. Don’t insert a timestamp or random request ID before the reference material, because it changes the beginning of the request every time. Keep tool definitions and their ordering stable too when the provider includes them in the cacheable prefix.

Check the cached-token fields the API reports before assuming savings. When a service charges for cache writes or storage, work out the effective cost of the complete reuse cycle instead of multiplying a headline cache discount by all input tokens. A stable prefix is worth having even without caching, because it makes prompt changes deliberate and reviewable. Any cache hits are a bonus on top.

References: OpenAI prompt caching, Gemini caching, vLLM prefix caching.