DeepSeek has open-sourced part of the technology it uses to run its models at scale. It is called DeepInfer. This is a walk through how DeepSeek's inference engine works, which optimisations it ships and why they matter so much for the performance and the cost of AI.
When people talk about DeepSeek, the conversation almost always ends up on the models: how many parameters they have, how well they code, how they score against Claude, GPT or Gemini. Another part of its technology seems just as interesting to me, and it gets far less attention.
How do you run a model with hundreds of billions of parameters, serve thousands of requests at once and keep the cost of inference competitive? Training the model is one job, and making it work well for thousands of users is quite a different one.
On 20 October 2026, DeepSeek released DeepInfer , its high-performance inference framework, under the Apache 2.0 licence. It comes from the system DeepSeek runs in production, although the open release does not yet include everything you need to stand up a complete service. It is worth reading well beyond the repository itself.
We recently wrote about why open models have never been this good and how DeepSeek is sharing several layers of its stack: the models, the inference engine and the software for building agents around them. DeepInfer is one of those layers, and it shows something that tends to get lost when models are compared: what a model can do and how well the infrastructure running it performs are two separate problems.
What an inference engine actually does
Picture an application that uses DeepSeek to analyse documents. A user uploads a twenty-page contract and asks which payment obligations it contains. Your application builds the context, calls the model and gets an answer back. From the outside, not much to it.
Between the request and the response sits a system that processes thousands of input tokens, spreads operations across devices, keeps state in memory and generates new tokens, almost always while it serves other requests.
The inference engine organises a large part of that work. It decides how the model runs on the hardware, how requests are batched, how memory is used and which techniques get more out of the same resources. Well-known projects in this category include vLLM, SGLang and TensorRT-LLM.
DeepInfer belongs to the same family, but it is tightly fitted to the architecture and operational needs of DeepSeek V4.1 Flash . DeepSeek states that it was built primarily on vLLM, with ideas taken from SGLang and TensorRT-LLM. What is interesting is how DeepSeek combines and specialises techniques that already existed.
That has an economic consequence: you can improve a model's performance without touching its weights. If you process more requests on the same GPUs, cut the time they spend waiting or use memory better, the system gains capacity without anyone training a smarter model. Your bill does not drop automatically because of it, but the cost of serving inference can change a lot.
Prefill vs decode: two different problems
Back to the contract.
The first thing the model does is process the document and the question: read the context and compute the representations it will use while generating. That phase is called prefill. Put simply, prefill processes the input tokens and builds intermediate state, including the key-value cache (KV cache) that lets attention computations be reused during generation. It parallelises well and, depending on context length, usually needs a lot of compute.
Then comes decode, where the model writes the answer one token at a time. Each token depends on the ones before it, so the output cannot be computed in one pass the way the prompt can.
The hardware needs change with it. In many configurations prefill is limited mostly by compute, and decode by memory access, data movement and concurrency. It is not a universal rule, but it explains why each phase calls for different optimisations.
Now picture hundreds of users at once. Some send huge documents, others two-line questions. Some expect short answers, others generate thousands of tokens. Handle that mix badly and you end up with GPUs waiting for data, blocked operations and users sitting through latency they should not have to. And GPUs are not cheap.
That is why one of the important decisions in large-scale inference is to optimise prefill and decode separately and, when it pays off, give them separate resources. DeepSeek does this in its production architecture. DeepInfer includes related mechanisms, although the open release does not yet ship the full phase disaggregation or the load balancing across replicas.
Keeping this distinction in mind makes the rest of the repository easier to read. Some optimisations aim to process more input, others to generate more tokens, others to cut the time the hardware sits idle. Each one targets a different problem.
Where each optimisation acts: prefill reads the prompt at once, decode writes token by token, and expert parallelism spans both phases.
Running a huge MoE model without GPUs waiting on each other
DeepSeek V4.1 Flash uses a Mixture-of-Experts (MoE) architecture. It has hundreds of billions of parameters, but at each inference step it activates only a few specialised experts for each token. That gives it a lot of capacity without using every parameter in every operation.
The trouble starts when those experts are spread across GPUs. If many requests hit the same experts while others are barely used, some GPUs saturate while others wait. On top of that, each token has to travel to the device holding its expert, be processed there and return to the model's flow. That trip costs time too.
DeepInfer uses expert parallelism to spread experts across devices and Expert Parallel Load Balancing (EPLB) to even out their use. According to DeepSeek's deployment notes, each GPU also holds a few redundant experts that replicate the busiest ones, and the placement is recomputed periodically from observed load. It also integrates DeepEP, built for efficient communication in MoE models, along with DeepGEMM, FlashMLA, TileKernels and other kernels tuned for specific operations. The goal is to get more out of the hardware and cut work that adds nothing to the answer.
One caveat matters here. A model that activates a small share of its parameters per token does not need only that share in memory. The infrastructure has to keep every expert that might be selected available and decide how they are distributed. Running a huge MoE still takes serious infrastructure, which is why parallelism, communication between GPUs and load balancing carry so much weight.
Simplified example: three tokens pick expert E1, GPU 1 saturates, and load balancing places a replica of E1 on GPU 4.
Three ways to split the work
DeepInfer lets you configure different parallelism strategies depending on hardware and workload. The official examples include these three:
python examples/offline_inference.py --ep 8
python examples/offline_inference.py --ep 2 --dp 8
python examples/offline_inference.py --ep 2 --cp 8They are three independent examples, not a sequence. Each needs a compatible environment with enough properly configured accelerators.
The first uses expert parallelism (EP) and spreads the model's experts across devices. It is central to MoE architectures, where each token may need different experts.
The second combines EP with data parallelism (DP), which splits work across replicas or execution partitions to handle different loads.
The third adds context parallelism (CP), which spreads the processing of the context across devices. It is especially useful for long sequences that consume a lot of attention resources.
The three strategies can be combined in many ways depending on hardware, model and workload. The hard part is finding a configuration that uses the resources without adding more communication and coordination than necessary. The same model can perform very differently depending on how you organise its execution, and that is one of the most interesting parts of inference at scale.
Keeping GPUs busy: the async scheduler
There is another optimisation I particularly like: asynchronous scheduling.
An inference engine does not spend all its time running the model. It also organises requests, prepares batches, manages state and coordinates execution, and part of that work happens on the CPU. In a decode step without overlap, the GPU sits idle while the CPU processes the current step's result and schedules the next one. If the GPU has to wait for the CPU, you are throwing capacity away.
DeepInfer's asynchronous scheduler overlaps preparing the next batches with running the current ones. While the GPU runs step N, the CPU is already scheduling step N+1 and post-processing the previous one. DeepSeek says it follows the approach of earlier implementations such as SGLang and NanoFlow, and it is enabled by default.
The repository calls it a zero-overhead batch scheduler. I would read the name as a design goal, because scheduling always costs something. What matters is cutting dead time and keeping the devices busy.
Improving a system's performance does not always require a faster GPU: there is often unused capacity in the gaps where the GPU waits on the CPU. Across dozens of accelerators, small efficiency losses on every step add up to a lot of wasted capacity.
DSpark: speculative decoding beyond one token at a time
DeepInfer also includes speculative decoding.
In conventional generation, the model produces a token, updates its state and moves on to the next. That sequential progress is one of the limits of decode. Speculative decoding proposes several candidate tokens at once and then checks them. If enough are accepted, the model advances several tokens in a single verification iteration and needs fewer steps.
DeepInfer does this with DSpark and a verification mechanism that uses confidence estimates to organise the work. Variable-length verification is turned on with DEEPINFER_ENABLE_MTP_VARLEN=1, and DEEPINFER_USE_NEXTN_DECODE sets the maximum draft length. The documented default is five.
Why does confidence matter? Because proposing tokens has a cost too. If you draft too many and then have to throw them away, that work is lost. Not every sequence is equally likely to be accepted, and speculating the same amount does not always pay off. According to DeepSeek's documentation, a confidence head estimates the acceptance probability of each draft token, and the scheduler picks which prefix is worth verifying to get the most effective throughput.
Confidence estimates help decide how to organise verification, but the proposed tokens are still checked.
Conceptual example: three draft tokens are accepted, the fourth is corrected and the fifth is never verified. Four tokens in one step.
The real gain depends on how many tokens are accepted, what it costs to produce and verify them, and the type of workload. Turning on DSpark does not make every request generate twice as fast. It is a good example of attacking a limit of the generation process without touching what the model can do.
SWA bounded replay: skipping work the decoder does not need
A less well-known optimisation in the repository is SWA bounded replay.
To follow it you need to know what sliding window attention is. Instead of letting every position always attend to the whole history, some attention mechanisms work only with a window of recent tokens. That removes part of the cost of long contexts, although the details depend on each model's architecture.
Now imagine the decoder needs to rebuild some state during prefill. A careless implementation could redo work over a sequence far longer than necessary. If the attention to be rebuilt is limited to a window, there is no need to replay the whole history.
Bounded replay keeps and reprocesses only a bounded suffix of the context, long enough to cover the dependencies that matter. DeepInfer documents it with two variables:
DEEPINFER_YOCO_PREFILL_TRUNCATION_MODE=compact
DEEPINFER_YOCO_PREFILL_KEEP_TOKENS=140The 140 tokens are the documented default for V4.1 and are not a general recommendation for any architecture. DeepSeek warns that the retained suffix must cover the attention window plus the maximum backtracking length the system may need. Keep less than that and you may end up changing the behaviour you expect from the model.
The saving is not small. According to DeepSeek's deployment notes, rebuilding that cache exactly would mean running the last 2,560 tokens through the 20 decoder layers; recomputing only the end of the prompt brings total prefill compute down by nearly half.
This technique solves a different problem from DSpark. Speculative decoding tries to move faster during generation, and SWA bounded replay avoids redundant work when rebuilding the context the decoder needs. They complement each other, and both make it clear that efficient inference depends on knowing the model you run in depth.
Reproducible outputs matter too
Some of DeepInfer's features are about control over results rather than tokens per second.
It can constrain generation with regular expressions, JSON Schema or EBNF grammars. If you extract data from invoices and your application expects JSON with certain fields, these constraints prevent a share of the format errors. The extracted values can still be wrong, because valid JSON can hold the wrong information, but at least you separate the format problem from the quality problem.
It also includes batch-invariant inference and deterministic sampling. When a service batches requests to make better use of its GPUs, some numerical operations return slightly different results depending on how the batch is composed, and those differences can end up changing the generated tokens. In a chatbot it hardly matters. If you are reproducing a bug, comparing two configurations or running a technical evaluation, consistent results help a lot.
DeepInfer has modes that keep results invariant to the batch and that control sampling with seeds. According to the current documentation, they work under specific conditions and some only on NVIDIA: deterministic sampling, for example, requires turning off DSpark's variable-length verification. Full reproducibility also depends on the model, the generation parameters and the runtime environment. What you gain here is a more controllable system.
DeepInfer vs vLLM and SGLang: what is different
Anyone who already works with inference infrastructure will reasonably ask why another engine is needed when vLLM, SGLang and TensorRT-LLM exist.
My answer is specialisation. vLLM and SGLang have broad ecosystems and support many model families. DeepInfer comes from infrastructure designed for DeepSeek's specific needs, which lets it make decisions closely fitted to the architecture of its models, the hardware they run on and the execution patterns it wants to optimise. The combination of specialised kernels, expert parallelism, load balancing, DSpark, asynchronous scheduling and SWA bounded replay comes out of that specialisation.
It would also be wrong to present all of this as DeepInfer's own inventions. Some pieces come from projects DeepSeek had already released separately; others use known techniques or build on earlier engines. The repository itself explicitly credits vLLM and the conceptual contributions of SGLang and TensorRT-LLM.
I see that as a strength. That is how open development works: some teams share technology, others reuse components, and each builds what it needs. Rather than asking whether DeepInfer will replace vLLM or SGLang, it is more useful to look at what can be learned from its decisions and which optimisations will end up benefiting the rest of the ecosystem.
There is not yet a public comparison controlled enough to claim that DeepInfer is faster than those engines in general. That would take the same model, on the same hardware, with equivalent configurations and representative workloads. The benchmarks DeepSeek has published do not make that comparison.
How to read DeepInfer's benchmarks
The repository includes prefill and decode performance tests, with scripts to reproduce the measurements on compatible setups. The requirements are steep: four NVIDIA SM100 GPUs for prefill and 64 for decode, each group within one scale-up domain. That is not something you put together on an ordinary server.
More than the hardware, what interests me is how to read the numbers.
For prefill, the benchmark measures tokens processed per second per GPU over intervals of steady execution. It first discards part of the warm-up and uses a defined measurement window. That shows how the engine responds as the amount of work it receives changes.
For decode, the documentation gives more to think about. The benchmark script computes throughput from the speed of the generation steps, the number of concurrent requests, the speculative tokens and the context parallelism:
throughput = avg_SPS × decode_num_requests × (1 + nextn) / cpavg_SPS is steps per second, decode_num_requests the decode batch size, nextn the configured speculative length and cp the context parallelism factor.
Here is the detail. The calculation counts every speculative token as accepted, whatever the real acceptance rate. The repository calls this full-acceptance accounting. With 64 requests and five speculative tokens, each step counts 384 tokens before the context parallelism factor is applied.
It is a useful convention for comparing runs under the same conditions, but those tokens need not have ended up in the final answer. To know the effective generation throughput you would have to count how many are actually accepted during verification.
The decode curve in the README uses a different convention, and the two should not be mixed. It is plotted with nextn=3 and an assumed per-token acceptance of 75%, which gives τ=3.25 tokens per step per request. That 75% is a parameter passed to the script that draws the chart, not a rate measured during the run. The curve describes how the system would scale if acceptance were at that level.
DeepInfer's original charts, cropped and in dark mode, with values and scales unchanged. The decode curve assumes 75% per-token acceptance.
Nor can the benchmark figure be carried over to the throughput of an application in production. Many more factors come in there: requests of varying length, shifting concurrency patterns, speculative acceptance that varies, input processing, memory management and all the infrastructure around the engine.
It helps to keep three things apart: the speed of the generation steps, the throughput as accounted, and the useful tokens actually produced. This does not take value away from DeepInfer's benchmarks. It sets their scope. It is very easy to see a tokens-per-second figure and assume it is the speed of an API, and often it is not.
Throughput, latency and utilisation are different metrics
When evaluating inference infrastructure, a company runs into numbers that look comparable but answer different questions.
| Metric | What it measures | Why it matters |
|---|---|---|
| Throughput | Amount of work processed per unit of time | Sets the aggregate capacity of the system |
| Latency | Time a request takes to start or finish its response | Directly affects the user experience |
| Utilisation | Share of the available resources actually used | Drives the economic efficiency of the infrastructure |
You can raise throughput by serving more requests at once and make each user's latency worse at the same time. DeepInfer's decode curve shows it: going from 8 to 256 requests per replica multiplies throughput per GPU while the speed each user sees drops to about a third. You can also improve hardware utilisation without saving in proportion, if the cost of the machines, their provisioning or the reserved capacity stays the same.
No single metric tells you which infrastructure is best for every application. A system that processes documents in the background will want to maximise total work. An interactive assistant will probably put time to first token and the generation speed the user perceives first.
DeepInfer brings useful optimisations for these problems, but which ones to use and how to configure them depends on the workload. That is why a technical benchmark needs to come with an explanation of what it measures.
DeepInfer is not yet a complete inference service
Before anyone tries to deploy it, one thing should be clear. DeepInfer is open source, but you cannot install it and offer a service equivalent to DeepSeek's official API the next day.
The public release targets offline inference and supports DeepSeek V4.1 Flash. It needs specific hardware and environments: the documentation lists NVIDIA SM100 GPUs and Huawei Ascend 950 accelerators, with their CUDA or CANN requirements.
DeepSeek also lists components that are not part of this release, including a complete KV cache system, operational prefill/decode disaggregation and load balancing across replicas. Those pieces become essential once you move from running a model to serving real traffic, with availability, scaling and maintenance requirements.
Releasing an engine is a long way from releasing an entire inference platform. You can have the weights, the code and very advanced optimisations and still need a team able to turn all of it into a reliable service. I do not say this as a criticism of DeepInfer: it gives a sense of how much work sits behind a model API.
Beyond DeepInfer
What I like most about DeepInfer is that it shows a part of the service that normally stays hidden behind an API: how experts are spread across accelerators, what happens when you try to generate several tokens in a single iteration, when it pays to avoid repeated computation or to separate the phases of inference, and why making a GPU wait less can matter as much as using a smaller model.
DeepSeek has not published a complete inference platform, but it has published part of the technical decisions that make one possible. That fits a broader trend. In recent years we have had extraordinarily capable open models, and now the tools to run them and build applications on top are starting to be shared too.
DeepSeek contributes to several of those layers with its models, with DeepInfer and with projects like DeepSeek Harness, which we take apart in our interactive microsite . They are independent pieces and you do not need to use them together, but they show that the model, the execution infrastructure and the software that turns it into an agent can each evolve on their own.
At Helmcode we work precisely at the infrastructure layer: we run open models in Europe and offer them through an API, so companies can use them without managing GPUs or maintaining their own inference systems. For teams with EU data residency requirements, that location is part of the point. None of this means we use DeepInfer internally. It interests us because projects like this widen the knowledge and the tooling available to the whole ecosystem.
For years, the conversation about open models has revolved around who scored best on a benchmark. It makes more and more sense to also look at how that result is achieved, what it costs to run and what infrastructure it needs. Part of the next improvements in open models will come from there, from running existing models better, and DeepInfer is one of the first places where you can read how that is done.
