DeepSeek releases open-source inference framework Deepinfer to cut self-hosting costs
DeepSeek launched Deepinfer, an open-source tool designed to reduce compute and energy costs for enterprises self-hosting AI models, currently limited to specific high-end hardware.
Chinese AI company DeepSeek released an open-source inference framework named Deepinfer on GitHub on October 21, 2026. According to a report by German tech outlet t3n, the tool aims to help enterprises that deploy and run large language models themselves to lower their computing and energy costs, making AI usage more affordable.
Deepinfer optimizes the inference process through three main technical approaches: distributing computational tasks more evenly across available processor cores; utilizing 'Speculative Decoding' to speed up response times; and better structuring model outputs before final delivery. Currently, the framework only supports DeepSeek's own V4.1-Flash model, with plans to expand to other proprietary models in the future.
It is important to note that Deepinfer is not intended for general consumers or web-based users, but rather for enterprise clients with high-performance server clusters. It requires underlying hardware equipped with Nvidia GPUs based on the SM100 architecture or Huawei's Ascend 950 NPUs. These chips are typically offered as part of complete server systems to enterprises and are expensive; for instance, a single Huawei Ascend 950 chip costs over $37,000. Consequently, this tool primarily serves large tech companies or cloud service providers with significant self-hosting capabilities.