Glossary
The 38 terms the guide relies on, in its own sense, each linked to the chapters that explain it.
No term matches. Try a shorter word.
- Acceptance rate
-
Accepted tasks divided by the tasks in the cohort. Report it next to cost per accepted task, since either number alone hides behavior. Loosening or tightening acceptance can move both.
In 02 Build a useful cost baseline 12 The economics of an AI feature
- Accepted task
-
A task whose result meets the requirements you defined for that operation. For extraction, for example: the right shape, consistent with business rules and correct against the source. Your product sets that bar. A successful API response alone doesn't meet it.
In 01 The true cost of an AI task 02 Build a useful cost baseline
- Allocated cost
-
A share of a fixed bill assigned to each task, such as a $1,000 monthly service divided by the tasks it handled. It changes with volume even when the bill does not. Compare with marginal cost.
In 01 The true cost of an AI task 11 GPU inference and deployment economics
- Batch API
-
A provider mode for requests that can wait, often hours, usually at a lower price than real-time calls. This is a pricing tier. Running work asynchronously in your own system and batching on a GPU are separate things.
- Break-even volume
-
The monthly volume at which a fixed-cost option costs the same as a metered one: the fixed fee divided by the variable saving per task. It only holds while capacity, quality and service stay comparable, so treat it as a scenario boundary.
In 11 GPU inference and deployment economics 14 When flat-rate inference makes sense
- Cascade
-
Starting with a cheaper model and escalating to a stronger one when a check finds a failure or a weak result. Worth it if failures are caught reliably and the checks are cheap.
In 04 Routing, cascades and reasoning budgets 03 Choose the model the task needs
- Chunking
-
How documents are split before indexing for retrieval. Chunks that are too small separate a rule from its exception; chunks that are too large pull irrelevant text into every request.
- Context compaction
-
Replacing a long transcript with a shorter working state: objective, constraints, findings, failed approaches and where the evidence lives. It costs tokens itself and can break prompt-prefix reuse, so do it when the reduction justifies it.
In 07 Agent memory, response caching and duplicate work 09 Agent loops, validation and retries
- Continuous batching
-
A serving-engine technique that interleaves requests of different lengths on the same GPU to keep it busy. It raises throughput on hardware you run and is unrelated to a provider's batch discount.
In 11 GPU inference and deployment economics 08 Batch and asynchronous inference
- Contribution margin
-
Net revenue for a task or an account minus the variable cost of serving it. Used per feature, it shows which usage patterns pay for themselves before company overhead is included.
- Cost per accepted task
-
Total attributable workflow cost divided by accepted tasks: inference, retrieval, tools, retries, compute, human correction and a share of fixed capacity. This is the guide's central metric: failed and abandoned work gets charged to the work that succeeded.
In 01 The true cost of an AI task 02 Build a useful cost baseline
- Decode
-
The phase in which a model generates output tokens one at a time. It stresses the hardware differently from prefill, which is why long inputs and long outputs cost differently to serve.
- Dedicated capacity also Fixed capacity
-
Inference capacity reserved for you and paid for by the month, whether a provider runs it or you run it on your own GPUs. The bill does not follow usage, so idle time is yours to pay for and growth arrives in steps.
- Distillation also Fine-tuning
-
Training a smaller model to do a narrow, stable task that a larger one handles today. It can lower the cost per call, and it adds training, evaluation, hosting and retraining costs. The license must allow it.
In 03 Choose the model the task needs 05 What open-weight models actually change
- Escalation rate
-
The share of requests a cascade sends on to the stronger model. Together with the cost of each check, it decides whether the cascade is cheaper than using the stronger model directly.
In 04 Routing, cascades and reasoning budgets 03 Choose the model the task needs
- Evaluation set also Eval set
-
A fixed set of real cases with known good outcomes, used to compare models or prompts before switching and to catch regressions after a change. Running it costs money, and that cost belongs in the decision.
- Flat-rate plan also Flat-rate inference
-
A recurring fee for a defined inference service, with model allowances and operational limits. No plan of this kind is unlimited, so check that your workload fits its allowances, rate limits, concurrency and latency.
In 14 When flat-rate inference makes sense 12 The economics of an AI feature
- Idempotency
-
The property that repeating the same intended operation has the effect of doing it once. It needs a durable operation identity, and for side effects such as payments or emails, a contract the downstream service supports. Different from response caching.
- KV cache
-
The attention state a request builds while reading its input, which every decode step then reads. It grows with context length and with the number of concurrent requests, which is why long contexts need more GPU memory.
In 11 GPU inference and deployment economics 06 Tokens, context and prompt caching
- Marginal cost
-
What one more task adds to the bill. On fixed capacity with room to spare it can be close to zero; once the service is saturated, the next task may require a new tier.
In 01 The true cost of an AI task 12 The economics of an AI feature
- Metered API
-
Inference billed by usage, usually per input and output token. Simple to operate and the bill follows the work, which suits variable traffic.
In 11 GPU inference and deployment economics 05 What open-weight models actually change
- Mixture of experts also MoE
-
A model architecture that activates only part of its parameters for each token. Serving it still needs memory for the total weights, so active-parameter counts understate what it takes to host.
In 05 What open-weight models actually change 11 GPU inference and deployment economics
- Open-weight model
-
A model whose weights are published for others to run, under a license that varies by release. Downloading costs nothing, but you still pay to run it, through an API, managed capacity or your own GPUs.
- P95
-
The value below which 95% of observations fall. Used for latency and for cost per task, because a few long runs can push the tail far above the median, and the average hides them.
In 01 The true cost of an AI task 11 GPU inference and deployment economics
- Payback period
-
One-time cost divided by monthly net saving. It only makes sense when the monthly saving is positive; one-time costs stay out of the monthly figure.
In 13 The optimization playbook 11 GPU inference and deployment economics
- Prefill
-
The phase in which a model reads the input before generating. Long prompts and retrieved context make it expensive; prefix caching can avoid repeating it.
In 11 GPU inference and deployment economics 06 Tokens, context and prompt caching
- Prefix caching
-
A serving engine keeping the KV cache of a shared prompt prefix and reusing it for later requests that start the same way. It saves the prefill for that prefix; every output token is still generated. On dedicated hardware the gain shows up as throughput rather than a smaller invoice.
In 06 Tokens, context and prompt caching 11 GPU inference and deployment economics
- Prompt caching
-
A provider feature that charges or processes a repeated input prefix differently. Writes, reads, TTLs and minimum sizes vary by provider, so check the cached-token fields in the response before counting savings.
- Quantization
-
Storing weights at lower precision, for example 8 or 4 bits instead of 16. It reduces memory and can raise throughput on supported hardware, but it can change model behavior.
In 11 GPU inference and deployment economics 05 What open-weight models actually change
- RAG also Retrieval-augmented generation
-
Retrieving relevant passages and sending them to the model with the question. Its cost spans ingestion, embeddings, storage, retrieval, reranking and every token of context it adds.
- Reasoning budget also Reasoning effort
-
A control some reasoning models expose to limit internal generation before the answer. Set it per operation; an output-token limit is not a substitute and can truncate the answer instead.
In 04 Routing, cascades and reasoning budgets 03 Choose the model the task needs
- Reranking
-
A second pass that reorders retrieved passages so only the most relevant reach the model. The extra pass has a cost, recovered only if it trims enough context or improves answers enough.
- Response caching
-
Reusing a valid earlier result for the same inputs, keyed by content and versions and scoped by tenant. It can save a whole call. Semantic caching, which matches similar rather than identical inputs, is riskier.
In 07 Agent memory, response caching and duplicate work 06 Tokens, context and prompt caching
- Retry amplification
-
Retries stacking across layers: an SDK that tries three times inside an application loop that tries three times can send nine requests upstream. Set one overall deadline and know each layer's policy.
- Routing
-
Choosing a model for each request. When the application already knows the operation, a static lookup is simpler than asking a model to route another model.
- Showback also Chargeback
-
Reporting AI spend to the team or feature that caused it. Chargeback goes one step further and bills it to their budget. Either needs costs tagged by team or feature first.
- Tokens per minute also TPM
-
A rate limit on how many tokens a key or account can process each minute. A monthly allowance says little about it, so check TPM, request rate and concurrency separately.
- Total cost of ownership also TCO
-
Everything it costs to run a service: inference or hardware, plus failover, network, monitoring, upgrades, security and engineering time, with one-time migration shown separately.
In 11 GPU inference and deployment economics 13 The optimization playbook