Qwen3.6 on an RTX PRO 6000 at full scale

This week we started moving our Qwen3.6-35B-A3B from FP8 to NVFP4. It sounds like a config change. It wasn't.

It ended in a GPU core dump, a cuda-gdb session, and a bug that turned out to live in cuDNN, not in vLLM. But the hassle paid off, because the model now serves faster and with more headroom than the FP8 build we came from, on the exact same hardware. Here is the whole thing, end to end.

Where we started: FP8

Our Qwen3.6 fleet runs one model per GPU. The baseline, and what every node except the canary was serving, is the FP8 checkpoint on vLLM 0.21 with the Triton kernels. This is our most-requested model, and with this setup it has, to this day, always served us well.

This is the reference number for the rest of the post, since it is the load test we use to measure the model's concurrency and scalability: 32 concurrent clients, ~81k tokens in, 256 out (vllm bench serve, random, --ignore-eos).

12.8k tokens/s aggregate, ~600 ms per output token, KV cache peaking at 55% under that load. That is the bar NVFP4 had to beat to be worth the trouble.

Why bother with NVFP4

The Qwen3.6-35B NVFP4 checkpoint from NVIDIA ModelOpt is a mixed-precision build: the MoE experts are 4-bit (W4A16_NVFP4), with FP8 elsewhere and an FP8 KV cache. Four-bit experts mean less VRAM spent on weights, which means more VRAM left for KV cache, which means more concurrent requests on the same card. On top of that, on Blackwell the narrower format should decode faster.

So the promise was: more throughput, more concurrency, the same GPU. Time to roll up our sleeves and try the change.

The canary crashed

We moved one node (call it the canary) to the NVFP4 checkpoint on vLLM 0.25.1 and gave it a small load test. Everything looked fine, so we put it back in the cluster to start taking real traffic. It started dying:

CUDA error: an illegal memory access was encountered
CUDA error: misaligned address (cudaErrorMisalignedAddress)

Roughly once an hour in production. Each crash killed the vLLM EngineCore, an external watchdog we keep watching the process restarted it, and every in-flight request in that window was dropped. Not production-safe. It blocked the whole fleet rollout.

The crash had a clear fingerprint:

• Only with CUDA graphs on. --enforce-eager ran clean for a 40-minute soak.
• Only at tensor-parallel 1 on this Blackwell card.
• Stochastic. Ordinary small batches, no correlation with prompt size, roughly one crash in tens of thousands of forward passes.

The config sweep (and a trap)

Before going deep, we swept the obvious knobs on the canary node, which we pulled out of traffic so we could experiment on it. This is simply a summary of the configs we tried:

Two things jump out. NVFP4 with CUDA graphs is genuinely faster than FP8 (~+11% tok/s, lower TPOT). And the "safe" answer, --enforce-eager, throws that win away and lands slower than FP8. The b12x MoE backend is a non-starter at this scale.

The last config fooled us. Disabling the DeepGEMM E8M0 scale path that vLLM bundles (VLLM_USE_DEEP_GEMM_E8M0=0) made the crash rare enough that a ten-minute load test ran clean. It looked like the fix. It wasn't: back in production, that same config still crashed, just less often. A short load test being green is not the same as being stable, and this is exactly the kind of bug that punishes you for believing it is.

Chasing the wrong kernel

Every Python traceback pointed at the Marlin MoE kernel (fused_marlin_moe). A careful read of the vLLM source even produced a plausible one-past-the-end write in Marlin's Stream-K barrier buffers, with a candidate one-line patch. It was wrong.

The tell is that CUDA errors are asynchronous: by the time the host sees "illegal memory access", the kernel that actually faulted is long gone and the error surfaces at the next stream sync, wherever that happens to be. The Marlin traceback was just where the bill came due, not where the money was spent. We needed to catch the faulting kernel red-handed.

The core dump

So we did. We built vLLM 0.25.1 from source with -lineinfo, ran it with GPU core dumps enabled, reproduced the crash under load, and opened the dump in cuda-gdb:

CUDA_ENABLE_COREDUMP_ON_EXCEPTION=1 CUDA_COREDUMP_FILE=/tmp/cuda_coredump_%h.%p.%t
# ... reproduce, then:
cuda-gdb -batch -ex "target cudacore /tmp/cuda_coredump_..." -ex "bt"

The dump named the guilty kernel, and it was not Marlin:

CUDA Exception: Warp Out-of-range Address
Kernel: cudnn_generated_fort_native_sm120_matMul_pointwise_pointwise_knob_3_128x128x128
Faulting instruction: STS.128 [R6+0xe910], R92

STS.128 is a 128-bit store to shared memory, at offset ~0xe910 (about 58 KB), past the block's allocated shared memory. The offset is a compile-time constant and shared memory is per-block, so this is not corruption fed in from somewhere else. The kernel writes out of bounds on itself. And it is a cuDNN runtime fusion-engine GEMM, not anything in vLLM.

Root cause

Here is why the whole fingerprint makes sense.

The workstation Blackwell GPU (sm120, our RTX PRO 6000) has about **99 KB of shared memory per block**. Datacenter Blackwell (sm100) has about 228 KB. cuDNN's fusion engine picks a "knob" (the tile size it chops the GEMM into) using a heuristic tuned for the big GPU, the datacenter one. On sm_120, with less memory per block, that tile's epilogue writes past what the block reserved.

• TP=1 produces the GEMM shapes that select the bad knob. TP=4 shards produce different shapes and dodge it.
• CUDA graphs capture that one bad launch config and replay it verbatim, so it faults deterministically on that shape. Eager mode reselects each call, so it usually lands somewhere harmless. That is why enforce-eager "worked" and why it was never actually safe.
• Stochastic because whether the out-of-bounds store hits a live allocation depends on the memory layout at that moment.

This is a known, fixed bug. The cuDNN 9.23.1 release notes put it plainly: "an issue where matrix multiplication with mainloop fusions could produce incorrect results on ... sm120 and sm121 has been fixed." torch 2.11.0+cu130 ships cuDNN 9.19.0.56, which predates the fix. There is a structurally identical sm_120 shared-store bug reported against llama.cpp too, so, as with almost everything in tech, we weren't the only ones who hit it.

The fix is one line

cuDNN 9.x is ABI forward-compatible, so you do not need to rebuild torch or vLLM. You just install a newer cuDNN into the serving environment:

pip install -U "nvidia-cudnn-cu13>=9.23.1"   # we run 9.24.0.43
python -c "import torch; print(torch.backends.cudnn.version())"   # -> 92400

That is it. No --enforce-eager, no Marlin patch, no CUDA-graph surgery. With cuDNN 9.24 and the exact config that used to crash on every load test at ~23 minutes, the node held a ~50-minute soak with zero crashes, then went back into production and has stayed there.

We posted the full core-dump evidence and this fix upstream on vllm-project/vllm#35566 , because a lot of people were (understandably) blaming vLLM for what is really an NVIDIA cuDNN bug that is already fixed in a later release.

The config that runs

For anyone doing the same thing, here is the serving config that is stable and fast on a single RTX PRO 6000 (sm_120), TP=1, 262k context:

python -m vllm.entrypoints.openai.api_server \
  --model /models/qwen3.6-35b-a3b-nvfp4 \
  --gpu-memory-utilization 0.95 \
  --max-model-len 262144 \
  --max-num-seqs 128 \
  --enable-chunked-prefill --max-num-batched-tokens 32768 \
  --enable-prefix-caching \
  --kv-cache-dtype fp8            # NOT nvfp4 KV, that crashes separately on sm_120
  # MoE backend: Marlin (auto), CUDA graphs ON (do not force enforce-eager)

# environment:
#   VLLM_USE_DEEP_GEMM_E8M0=0     # avoids a separate scale-format corruptor
#   nvidia-cudnn-cu13 >= 9.23.1   # the actual fix

The two things that matter most: cuDNN >= 9.23.1 (the fix), and keep the KV cache in FP8, since NVFP4 KV has its own separate crash on sm_120.

NVFP4 vs FP8, the payoff

Back to the numbers, the same load test (32 concurrent, 81k tokens in / 256 out), now comparing the FP8 baseline against the stable NVFP4 config:

Same 96 GB card, same 32-way concurrency, same 81k-token prompts. NVFP4 decodes faster (lower TPOT is the big one for a coding assistant sending lots of small turns), pushes ~11% more aggregate throughput, and, because the 4-bit experts free up VRAM, the KV cache peaks 12 points lower. That KV headroom is real concurrency headroom: room to push past 32 clients before the cache saturates.

A note on first-token latency: with 81k-token prompts, TTFT is dominated by prefill and lands in the tens of seconds regardless of format, since prefill compute is similar. For normal community-sized prompts, first token is about a second. The NVFP4 win is in decode throughput and memory, which is exactly what we want on a node that is going to serve inference at scale.

What we shipped

The canary is back in production on NVFP4, and the fleet rollout to the other Qwen nodes is a straightforward cuDNN bump per node. Along the way we also taught our watchdog to notice a silent engine wedge (the API answering 200 while the engine makes zero progress), which the old "can I GET /v1/models?" probe was blind to, but that is a story for another post.

The lesson we take away: an async CUDA error will lie to you about where it came from, a green ten-minute load test will lie to you about stability, and the only thing that did not lie was the core dump. When a GPU bug is stochastic and points everywhere, stop reading tracebacks and catch the faulting instruction directly.

More open models, on hardware we actually understand. Onward. 🚀