DeepSeek Open-Sources Internal Inference Framework DeepInfer for High-Throughput Deployment

Chinese AI company DeepSeek announced on October 20 that it is fully open-sourcing its internal inference framework, DeepInfer, along with its companion token encoding/decoding component, deepseek-recipe. This follows previous releases of model weights, the TileLang compiler tool, and inference operator libraries, marking a further opening of infrastructure-level assets.

The inference framework serves as a critical middle layer connecting model capabilities to actual services, responsible for orchestrating massive compute clusters. Its efficiency directly determines the compute cost and power consumption of AI services. DeepSeek stated that by sharing internal deployment experience, they aim to help developers and enterprises optimize throughput and latency for large-scale inference, thereby efficiently building private and public cloud deployments, reducing compute costs, and making intelligence accessible at lower prices to a broader audience.

According to the announcement, DeepInfer integrates high-performance operators with a deeply optimized inference engine, supporting distributed inference, speculative decoding, SWA Bounded Replay (Sliding Window Attention Bounded Replay), and structured output. It explicitly supports both NVIDIA and Huawei Ascend hardware platforms. DeepSeek has already implemented high-throughput, low-latency large-scale inference services for DeepSeek V4.1 Flash based on this framework. The simultaneously open-sourced deepseek-recipe is a production-grade token encoding/decoding component fully aligned with official API processing logic, ensuring the community receives a consistent API experience.