DeepSeek Open-Sources DeepInfer, Its Production Inference Framework Supporting Nvidia Blackwell and Huawei Ascend
DeepSeek has open-sourced DeepInfer, its production-grade inference framework featuring speculative decoding and prefill optimizations, with support for Nvidia Blackwell and Huawei Ascend 950 chips, though some core scheduling code remains closed.
ImportanceMaterialEvidenceE2 unreplicatedWrite-upQuick
Chinese AI startup DeepSeek on October 20, 2026, open-sourced DeepInfer, the inference framework powering its own production services. The framework aims to solve the challenge of serving large models to a massive user base at a reasonable cost. According to DeepSeek, one cluster currently serves 155 trillion tokens per day, generating approximately $38 in revenue per GPU per hour at off-peak pricing.
For now, DeepInfer supports only a single model, DeepSeek-V4.1-Flash, defaulting to the updated 1017 version released over the weekend. The framework is compatible with Nvidia Corp.'s SM100 Blackwell GPUs and Huawei Technologies Co. Ltd.'s Ascend 950 neural processing units, but does not support Hopper or other older Nvidia architectures. Two features ensuring identical outputs across runs work only on Nvidia hardware. All published performance benchmarks were run on Nvidia chips, and the figures are self-reported by DeepSeek.
Technically, DeepInfer employs a speculative decoding system called DSpark, where a draft model proposes five tokens at a time and a confidence head predicts which ones the main model will accept. This reduces the number of draft tokens checked per step by about 30%, yielding an estimated 4% to 7% throughput gain at production load. A second technique claims to nearly halve total prefill computation by running only the last 128 tokens of each prompt back through the model's decoder layers.