Chapter 10: Tools, search and RAG

Overview#

A documentation assistant receives a question, searches a vector database and sends twenty passages to the model. The first two contain the answer; the rest are related but unnecessary. The search may be cheap, but the model still processes every returned token.

RAG costs span ingestion, embeddings, storage, retrieval, reranking, added prompt context and the final model response. Paid web search, OCR and other tools can add separate charges. Optimizing only the vector query or only the generation model misses how the two interact.

Practical#

Account for query-time and ingestion-time costs separately. A mostly static knowledge base receiving millions of questions has different economics from a frequently updated archive receiving few queries.

Don’t treat top_k, the number of retrieved passages sent to the model, as a universal quality knob. At 500 tokens per passage, twenty passages add roughly 10,000 input tokens; five add 2,500. At $2 per million uncached input tokens, the difference is $0.015 per request, or $1,500 across 100,000 requests. If the answer depends on passage eight, though, cutting blindly to five is a false economy.

Another option is to retrieve broadly and then filter with reliable metadata, hybrid search or reranking before sending evidence to the generation model. Reranking has its own cost and pays off only if it cuts enough context or improves quality enough.

Not every task needs retrieval. Reformatting the answer you just produced probably doesn’t require another vector search, and a known order status may be better obtained directly from the authoritative application API than inferred from documents.

To set top_k from evidence, log two things per request: the passages retrieved, with their rank, and the passages the answer actually used. Citations in the answer give you the second list cheaply; for answers without citations, review a sample by hand or with a grader. From those logs, track the rank of the deepest passage each answer needed, grouped by query type. If policy questions almost always draw on the top three passages while troubleshooting questions often reach rank eight or nine, give each type its own top_k instead of sending ten passages to everything. Recheck the numbers after any change to chunking, the embedding model or the reranker, because rank positions shift.

Ledgerline. The research agent sent 20 retrieved passages with every question. Logging which passages the reports actually cited showed that most query types never used anything past the eighth. With top_k set per query type, and the broad ones still allowed 20, research falls from $640 to $560 a month.

Deep Dive#

Chunk size affects answer quality as well as cost. Tiny chunks can separate a definition from its exception, and large ones can pull in irrelevant sections. Preserve headings, tables, code blocks and document versions where those boundaries matter.

If the answer is wrong, inspect what was retrieved before upgrading the generation model. A larger model can’t reliably reason from a policy paragraph that never reached it. Poor retrieval also tends to trigger extra searches and expensive retries.

A small, stable reference set reused by every request may be cheaper as cached prompt context (Tokens, context and prompt caching) than as a separate retrieval workflow. A large or frequently updated collection usually benefits from selective retrieval.

Embedding costs#

Embeddings are paid at two moments. Ingestion embeds every chunk of the corpus; query time embeds each incoming question. Query embeddings are short and grow with traffic. Ingestion embeddings grow with corpus size and with how often you rebuild.

Vectors from different embedding models can’t be compared, and new chunk boundaries produce new chunks, so changing either the embedding model or the chunking means re-embedding the whole corpus. Each rebuild is a one-time cost, but it comes back every time you change one of them. Version the index by embedding model and chunking configuration, build the new version alongside the old one, and switch traffic only after it passes your retrieval checks.

Illustrative: a corpus of 2 million chunks at 400 tokens each is 2,000,000 × 400 = 800 million tokens to embed on every full rebuild, before any queries.

Storage and index size grow with chunk count and vector dimensions. A float32 vector takes 4 bytes per dimension.

Illustrative: at 1,536 dimensions, one vector is 1,536 × 4 = 6,144 bytes, and 2 million chunks need 6,144 × 2,000,000 ≈ 12.3 GB of raw vectors before index overhead. At 768 dimensions the raw size halves to about 6.1 GB.

Run ingestion embeddings in large batches rather than one request per chunk; if your provider offers discounted asynchronous processing, ingestion is a natural fit (Batch and asynchronous inference). Store a content hash per document or chunk and embed only what changed since the last run, so a daily update doesn’t re-embed the unchanged majority.

Smaller embedding models are often enough for retrieval and lower ingestion, query and storage costs together. Don’t assume either way: run your own evaluation set (Choose the model the task needs) and compare how often the passages the answer needs land in the top results.

References: OpenAI web search, OpenAI file search, LangChain retrieval.