Chapter 11: GPU inference and deployment economics

Overview#

A GPU instance can look inexpensive when you divide its hourly price by theoretical token throughput. That calculation assumes the hardware is busy producing useful work most of the time, which real traffic rarely allows.

A model may fit in GPU memory but struggle with concurrent long-context requests. Traffic may arrive in peaks, and strict latency requirements may force you to keep unused capacity available. Self-hosting also makes you responsible for operating the service.

Setting a GPU rental listing next to an API pricing page tells you little. Compare options that deliver the same outcomes at the same service level.

Practical#

Describe the workload before selecting infrastructure: input length, output length, concurrency, latency objectives, traffic peaks and availability requirements. Track time to first token, generated-token rate, queue time, tail latency and completed accepted tasks.

For a dense model with 32 billion parameters, raw weight storage is approximately 64 GB at 16-bit precision, 32 GB at 8-bit and 16 GB at 4-bit, before quantization metadata, runtime buffers, KV cache and other overhead. For a mixture-of-experts model, plan memory for the total weights, not just the number activated per token.

Compare three options:

Say a deployment costs $6,000 monthly and a metered API costs $0.035 per accepted task. If self-hosting adds $0.003 variable cost per task, the simplified break-even volume is:

This holds only while the deployment can handle that volume without adding capacity and both options deliver equivalent quality and service. Real GPU costs often increase in steps as you add replicas.

Ledgerline. Ledgerline priced a dedicated replica for extraction and stopped quickly. At 40,000 invoices a month, the whole extraction line costs $567, while a single replica in the chapter’s example costs $6,000 before any traffic. The metered API stays until volume is several times higher, and the worksheet makes it easy to rerun the sums when it is.

Deep Dive#

During prefill, the model processes the input sequence. During decode, it generates output tokens. Scheduling and memory pressure differ between the two phases. Continuous batching can improve useful throughput by coordinating requests of different lengths; paged KV-cache management reduces wasted memory allocation; prefix caching can reuse the KV cache of a shared prefix across requests and skip repeating its prefill (see Tokens, context and prompt caching).

Quantization can reduce weight storage and sometimes improve throughput on supported hardware, but it can change model behavior and doesn’t make generation free. A smaller weight footprint helps only if weight memory is what limits the deployment.

On the $6,000 monthly deployment from the Practical example, 100,000 accepted tasks allocate $0.06 per task. Handling 160,000 on the same hardware allocates $0.0375. The invoice doesn’t change, but each task costs less. Pushed too far, higher utilization worsens queueing and P95 latency, so a reliable service may need some spare capacity.

Include failover, network, monitoring, upgrades, security and allocated engineering time in total cost. If migration costs $24,000 and saves $2,000 monthly in steady state, simple payback is 12 months before other risks and changes.

Comparing GPU purchase options#

Cloud GPU capacity is usually sold in three forms. On-demand capacity costs the most per hour and can be released at any time, though the GPU type you want may not be available when you ask for it. Reserved or committed capacity costs less per hour in exchange for a term, and you pay for it whether traffic arrives or not. Spot or preemptible capacity is cheaper still, but the provider can reclaim it with little notice.

That interruption risk decides where spot fits. Batch jobs, offline evaluation and other retryable work can absorb a reclaimed node by resuming from a checkpoint or requeueing. Latency-critical serving usually can’t, unless on-demand or reserved capacity covers the peak by itself.

Price whole nodes. A GPU-hour figure leaves out CPU and system memory, storage for model weights, networking and data transfer. Then convert the node price to cost per accepted task at the throughput you measured for your workload. Two GPUs can swap places in the ranking once throughput and batching are included. Prices and discount structures change often, so check each provider’s current pricing pages, such as the AWS EC2 pricing linked below, before running the comparison.

Capacity break-even as a range#

Keep demand shape and financial assumptions in one worksheet:

At 180,000 tasks, the metered alternative is 180,000 × $0.035 = $6,300/month. Dedicated is $6,000 + 180,000 × $0.003 = $6,540/month: more expensive, even before migration. At 190,000 tasks, metered is $6,650 and dedicated is $6,570. At 220,000 tasks, though, the fixed deployment in this example runs out of capacity, so you can’t extrapolate the apparent saving without adding another replica or changing the service requirement.

Put together, dedicated capacity in this example wins only in a narrow band, from 187,500 to 190,000 tasks a month, before a second replica is needed. Below the band the metered API is cheaper, and above it the second replica adds another $6,000 of fixed cost.

Monthly average traffic doesn’t establish capacity on its own. Size for peak arrival rate, input and output length distribution, per-request context, concurrency and P95 latency. A quote for dedicated capacity or a flat-rate plan should spell out throughput and queue behavior as well as monthly token allowances.

When comparing a migration, keep recurring net savings separate from the one-time cost; The optimization playbook covers payback. Treat any break-even point as the edge of one scenario.

Before committing to dedicated capacity, rerun the worksheet with your own volumes, peak load and latency targets. The calculator below starts from the figures in the table; move the volume past one replica’s capacity to see the step.

The same calculation, with a volume table, is in the downloadable worksheet.

References: vLLM, TensorRT-LLM, AWS EC2 pricing.