Why DeepInfer Is Suddenly Everywhere

Why it’s suddenly everywhere

1. 01

   On October 20, 2026, DeepSeek open-sourced DeepInfer on GitHub under Apache 2.0 — the production engine that serves one cluster’s 155 trillion tokens per day — and the repo passed 12,700 stars within a day.

2. 02

   The timing was one day after the U.S. Department of Homeland Security ordered federal civilian agencies to stop buying DeepSeek products and strip them from existing systems; researchers read the release as “Washington shuts the door, DeepSeek opens the engine room.”

3. 03

   It runs on exactly two platforms: NVIDIA’s Blackwell-generation SM100 and Huawei’s Ascend 950 — skipping all older NVIDIA GPUs and signaling a route around U.S. export controls.

4. 04

   DeepSeek disclosed unusually candid production economics in a deployment Q&A: 96% of input tokens hit cache, first token lands in ~400 ms, and each GPU earns about $38 per hour at valley electricity pricing.

The timeline

1. 2024-12The model drops, the engine stays hidden

   DeepSeek-V3 (and R1 weeks later) proves frontier-class weights can be open; how DeepSeek actually serves them cheaply remains a black box.

2. 2025-02Open Source Week

   Five repos in five days — FlashMLA, DeepEP, DeepGEMM, parallelism strategies, and the 3FS file system — the component layer, but not the engine that stitches them together.

3. 2025-04The promise

   DeepSeek publishes "The path to open-sourcing the DeepSeek inference engine": its serving stack is a heavily rebuilt early fork of vLLM, too entangled with internal infrastructure to release yet.

4. 2026-10-17DeepSeek-V4.1-Flash ships

   A faster snapshot of the V4 line lands on Hugging Face under the MIT license, with a 1M-token context — the model the new engine is built around.

5. 2026-10-19Washington acts

   DHS issues a FASCSA exclusion and removal order against DeepSeek for U.S. federal civilian agencies; China’s commerce ministry objects the next day.

6. 2026-10-20DeepInfer goes public

   DeepSeek open-sources DeepInfer (Apache 2.0) plus the deepseek-recipe token codec (MIT), posts a Zhihu technical essay and a deployment Q&A, and its GitHub tops 12.7k stars in 24 hours.

The backstory

What an inference engine actually is — and why this one mattered

A trained model is a pile of weights. The inference engine is the machinery that turns those weights into a service: it batches incoming requests, schedules them across GPUs, caches what has already been computed, and ultimately decides how fast the answer arrives and how much each token costs in silicon and electricity. It is the least visible layer of an AI company and, for a lab selling API access, arguably the one closest to the profit margin. Nobody outside DeepSeek had seen theirs — only its effects, in the form of API prices competitors struggled to match.

The release on October 20 changed that. DeepInfer is the framework DeepSeek says powers its own production service for DeepSeek-V4.1-Flash. The design’s centerpiece is disaggregation: a single request is split into two phases with opposite appetites. The prefill phase reads your whole prompt at once and craves raw compute; the decode phase then generates the answer token by token and spends most of its time reading parameters and cache from memory. Instead of forcing one machine to do both jobs badly, DeepInfer puts prefill on conventional 8-GPU nodes and decode on 64-to-128-GPU “supernodes” — and DeepSeek says its own efficient minimum deployment is 64 GPUs for prefill plus 64 for decode.

The 96% trick

The single most striking number in the documentation is not a benchmark — it is 96%, the share of incoming tokens DeepSeek says hit cache and never need recomputing. Agent-style workloads are pathological about repetition: every tool call resends the system prompt, tool descriptions and the entire conversation so far. DeepInfer stores the computed KV cache in a 3FS storage cluster of roughly 100 PiB of solid-state drives — agent traffic for three days, chat traffic for seven — so that any prefill machine can reuse any prior prefix. The payoff is brutal: a prefill GPU processes about 60,000 uncached tokens per second, but 1.5 million per second once cache hits are counted, a 25-fold gap. Even the misses are cheapened: because V4.1-Flash leans on sliding-window attention, a cache hit only needs the last 128 tokens replayed, which DeepSeek says cuts nearly half of total prefill compute.

A 19-month-old promise, kept on a geopolitical beat

DeepSeek first said it would open-source this engine in April 2025, in a note explaining why it could not yet: the codebase had diverged too far from the early vLLM fork it began as, and the parts that made it fast were lashed to internal infrastructure. The October 20 release keeps the letter of that promise with an honest asterisk. The core engine is public — optimized kernels, expert parallelism with load balancing, DSpark speculative decoding that skips low-probability draft tokens, asynchronous scheduling — but the KV cache system, the prefill/decode disaggregation glue and the load balancers are not, precisely because they remain coupled to internal systems. What DeepSeek released instead of that code is the design itself, in a long technical essay and deployment Q&A.

And then there is the context no one can ignore. On October 19, the U.S. Department of Homeland Security ordered federal civilian agencies to exclude and remove DeepSeek products under FASCSA supply-chain authority, calling the models a bundle of “threat vectors.” One day later DeepSeek responded not with a statement but with a codebase — one that runs on Huawei’s Ascend 950 as a first-class platform alongside NVIDIA’s newest chips, and not on older NVIDIA hardware at all. DeepSeek researcher Deli Chen framed the moment in openly ideological terms, casting open versus closed source as a contest between two futures for AI. Whether you read that as principle or positioning, the effect is the same: the cheaper-to-run blueprint is now public, on both sides of the chip divide.

Key facts

Released

2026-10-20

License

Apache 2.0

GitHub stars

12.7k in 24h

Serves

155T tokens/day

Cache hit rate

~96% of input

Hardware

SM100 + Ascend 950

What people are saying

Common questions

Can I run DeepInfer at home or on a small server?

Realistically, no. It supports exactly one model (DeepSeek-V4.1-Flash) on exactly two platforms — NVIDIA SM100 data-center GPUs and Huawei Ascend 950 NPUs — and DeepSeek’s own guidance treats 64 prefill plus 64 decode GPUs as the smallest efficient unit. This is infrastructure code for operators and cloud providers, not a hobbyist tool.

If key parts are missing, what is actually useful here?

Three things: the offline engine with DeepSeek’s production kernels and scheduling tricks; deepseek-recipe, the MIT-licensed token codec that normalizes OpenAI and Anthropic API formats; and the documentation, which amounts to a rare engineering blueprint of how a frontier lab serves 155 trillion tokens a day. For researchers, the design notes may matter more than the code.

Is this connected to the U.S. government ban?

Officially, DeepSeek announced the open-sourcing as a cost-and-energy initiative. But the sequence — a DHS exclusion and removal order on October 19, the engine going public on October 20 — was read across the industry as a deliberate countermove: shut out of U.S. government systems, DeepSeek made its serving stack freely available to everyone else, with first-class support for Huawei chips.