GPUs for AI

Choosing GPUs for model inference is not about buying the most expensive one. It is about buying the one that understands the format your model lives in.

Every GPU generation adds hardware support for smaller number formats (FP8, NVFP4, FP6), and each new format means one of three things: bigger models in the same VRAM, more throughput, or fewer GPUs to serve the same load.

But not every GPU supports every format. And if your model uses a format the GPU does not accelerate in hardware, you pay the penalty of the software-emulated path, which is usually 2x to 10x slower.

In this post we walk through the NVIDIA GPU families that matter today (July 2026), the precision formats each one accelerates in silicon, and how to translate that into buying decisions: which GPU you need to serve anything from a compact MoE like Qwen3.6-35B-A3B up to a massive one like GLM-5.2.

1. The map of architectures

NVIDIA: the line of succession

NVIDIA has shipped six datacenter architectures in six years. Each one introduces new-generation Tensor Cores that understand smaller precision formats.

One detail people often get wrong: the RTX PRO 6000 is Blackwell, not Ada. It shares the B200's 5th-gen Tensor Cores (NVFP4, FP6, FP8 in hardware), with 96 GB of GDDR7. Don't confuse it with the old "RTX 6000 Ada Generation" (48 GB, no FP4). That difference is exactly why we run NVFP4 in production, as we covered in the previous post .

2. Precision formats: what each GPU accelerates in hardware

Not every number format runs the same on every GPU. A model quantized to NVFP4 does not automatically run at full speed on an H200, because Hopper has no 4-bit Tensor Cores: it emulates them over FP8/FP16 and you lose the whole throughput and memory advantage.

One important clarification, because it is a common mistake: INT8 is native on the Tensor Cores since Ampere (A100). It is not emulated. What changed with the latest generations is not INT8, it is 4-bit: INT4 peaked with Ampere, Hopper set it aside, and Blackwell replaced it outright with NVFP4 (which keeps far more numerical precision at the same size thanks to per-block scaling).

The table that matters

Practical implications

• Hopper (H100/H200) is the queen of FP8. If your model is in FP8, and today most open-source models ship official FP8 checkpoints, Hopper serves it at full speed with its Transformer Engine. What Hopper does not understand is 4-bit floating point.
• Blackwell unlocks FP4 and FP6 in hardware. That NVFP4 NVIDIA introduced in Blackwell is not just another quantization: they are 5th-gen Tensor Cores that operate natively at 4 bits. The NVFP4 checkpoint of Qwen3.6-35B-A3B we run in production takes half the VRAM of FP8 and, as we described in the previous post , delivers +11% throughput and 12 points less KV cache usage on the same RTX PRO 6000.
• Hopper does not understand FP4 or FP6. If you load an NVFP4 checkpoint on an H200, the GPU runs it with software kernels over FP8 or FP16, losing the whole throughput advantage. The model works, but you gain nothing.

3. Which model fits in each GPU?

This is the exercise every company should do before buying. Let's take the three models we run in production and see which GPU serves them.

VRAM required per model and precision

✅ = Fits in a single current GPU (accounting for KV cache and serving overhead). The pocket rule: FP16 ≈ 2 bytes/param, FP8 ≈ 1 byte, NVFP4 ≈ 0.5 bytes. Then add the KV cache, which grows with context and batch.

What this means for buying decisions

• Qwen3.6-35B-A3B is tiny in VRAM. In NVFP4 it takes ~18 GB: it fits in any current GPU, even an RTX PRO 6000 (96 GB) with plenty of room for KV cache and concurrency. It is exactly the model we serve on that card.
• DeepSeek-V4-flash (284B total, 13B active) changes category with the format. In NVFP4 ~142 GB fits comfortably in a single B200 (192 GB) or B300 (288 GB); in FP8 ~284 GB you need a B300 or two B200. Even though only 13B activate per token, all 284B must be loaded in memory because any expert can be invoked.
• GLM-5.2 (753B total, 40B active) is the extreme case. In FP8 ~753 GB you need 4 B200 or 3 B300. In NVFP4 ~377 GB you drop to 2 B200 or 2 B300. That is why format efficiency is critical in large MoEs: it does not reduce compute per token, but it does reduce the number of GPUs you need just to hold the model loaded.

4. Throughput: tokens per second

Having a model that fits is not enough. You have to serve requests at a useful speed. This is where memory bandwidth (HBM) and Tensor Cores make the difference.

Inference throughput (~70B reference, FP8, aggregate with batching)

The real case: Qwen3.6 NVFP4 in production

We validated these figures with real production data. Migrating our Qwen3.6-35B-A3B checkpoint from FP8 to NVFP4 on an RTX PRO 6000 (96 GB, Blackwell sm_120) gave us:

The full write-up of the migration, including the cuDNN bug we hit along the way (a kernel that picked datacenter sm_100 tilings and wrote out of bounds on sm_120's 99 KB of shared memory), is in the previous post . The summary: same hardware, more throughput, just by changing the checkpoint format.

5. What to rent/buy for your use case?

There is no best GPU. There is the right GPU for your workload.

🧪 Research and experimentation

If you are doing fine-tuning, R&D, or proofs of concept:

• H100. Enough for almost everything and the natural choice for intermittent use.
• RTX PRO 6000 (Blackwell, workstation). Local capacity without depending on the cloud, with native NVFP4 and 96 GB.

🚀 Production at scale (compact models)

Serving a compact MoE like Qwen3.6-35B-A3B:

• H200: 141 GB of VRAM give you room for long KV cache (256K contexts) without breaking a sweat. Still the FP8 workhorse.
• B200 in NVFP4: if your model has an NVFP4 checkpoint, the B200 gives you the highest throughput per GPU.
• RTX PRO 6000: native NVFP4 with 96 GB; enough to serve models of this size locally.

🐘 Massive models (DeepSeek-V4-flash, GLM-5.2)

• B200 cluster (2-4 GPUs): with NVFP4, DeepSeek-V4-flash fits in 1-2 cards and GLM-5.2 in 2. Without NVFP4 you need twice as many.
• B300 (Blackwell Ultra): 288 GB per GPU, so you need fewer GPUs for the same model thanks to its VRAM.
• H200 cluster: a solid alternative if the model is in FP8.

⚡ Ultra-low latency (coding agents, real-time chat)

For coding agents that send many small requests and need fast responses:

• B200 / B300: the lowest TPOT (time per output token) thanks to 8 TB/s of bandwidth makes each token arrive sooner.
• H200: also excellent if the model is FP8.
• Avoid A100 for low latency: its bandwidth (~2 TB/s) is an obvious bottleneck in the decode phase.

🖥️ Mixed workload (AI + graphics + virtualization)

• L40S / RTX 6000 Ada: they combine Tensor Cores with RT Cores (ray tracing) and video output. For engineering, design, or virtualization (vGPU) workstations, they are still the natural choice. If you also want native FP4 on the desktop, the RTX PRO 6000 (Blackwell) is the evolution.

6. What's coming

Blackwell Ultra (B300 / GB300), already available

• 288 GB HBM3e per GPU (8 TB/s), thanks to 12-high stacks
• 15 PFLOPS FP4 dense per GPU
• GB300 NVL72: 1.1 ExaFLOPS FP4 per rack (1.5x GB200 NVL72)
• 1,400 W TDP (liquid cooling mandatory)

In practice: a ~200B parameter model in NVFP4 fits in a single GPU, and for a massive one in FP8, 2-3 B300 give you plenty of room even for long KV cache.

Vera Rubin (R100), in production H2 2026

• TSMC 3 nm, 336B transistors (two dies)
• HBM4: 288 GB per GPU, 22 TB/s of bandwidth (2.8x over Blackwell)
• 50 PFLOPS FP4 inference per chip
• Rubin NVL144: the new rack reference, ~3.3x the performance of GB300 NVL72

NVIDIA entered production in June 2026, with cloud availability (AWS, GCP, Azure, Oracle) rolling out through the second half of the year.

7. Conclusion: how to make the decision

For any company evaluating GPUs today, this is the recommended process:

1. Pick your target model. Not "a GPU", but "I am going to serve GLM-5.2 at 200 req/s".
2. Pick the inference precision. FP8 is the standard today; NVFP4 is the competitive edge if your hardware supports it.
3. Compute the required VRAM. Model weights + KV cache (context x batch) + serving overhead.
4. Pick the GPU with native Tensor Cores for that precision. NVFP4 on Blackwell is not the same as NVFP4 emulated on Hopper.
5. Compare real throughput (tokens/s) and effective VRAM, not catalog specs.
6. Add a 20-40% margin for traffic spikes and growth.

The right GPU lets you serve more model with less hardware. The wrong one ties you to a format, a platform, or a VRAM capacity that limits what you can serve. The difference is understanding what each generation accelerates in silicon and what it does not.

At Helmcode we have processed more than 574 billion tokens, with peaks of ~14 billion tokens a day. The three models in this post (Qwen3.6-35B-A3B, DeepSeek-V4-flash, and GLM-5.2) are exactly the ones we serve most; the numbers are open on our public data page . We have gone through A100, H100, Ada, and now Blackwell, and every generational jump has given us more capacity without changing the number of servers. Because optimization is not only the software: it is knowing which hardware understands your model.

Need GPUs for your workloads? Talk to us and we'll tell you about our availability of dedicated Blackwell GPUs that we can rent to you and/or serve you directly from our inference API.