DeepSpeed-MoE: Addressing Inference Bottlenecks in Mixture-of-Experts Models via Architectural and System Optimizations

In the development of large language models, the training costs of dense models rise sharply as parameter counts increase, making simple model scaling unsustainable due to hardware resource limitations. Mixture-of-Experts (MoE) models, as a sparse architecture, can significantly reduce computational costs during training; however, their massive parameter count leads to a surge in memory requirements and low inference efficiency, limiting their practical application. The DeepSpeed-MoE paper addresses this pain point by proposing an end-to-end solution aimed at balancing model quality, training cost, and inference efficiency.

One of the core contributions of this research is expanding the scope of MoE applications and proposing new architectural designs. Traditional MoE models are mostly used for encoder-decoder tasks, but this paper successfully applies them to autoregressive natural language generation (NLG) tasks, demonstrating that a 5x reduction in training cost can be achieved while maintaining equivalent model quality. To address the issue of low "parameter efficiency" in MoE models, the authors designed Pyramid-Residual MoE (PR-MoE). This is a hybrid dense-sparse architecture that applies sparsity only where experts are most effective through residual connections, thereby reducing the parameter size of MoE models by up to 3x without changing model quality or significantly altering computational requirements. Furthermore, using staged knowledge distillation techniques, the authors created a distilled version of PR-MoE called Mixture-of-Students (MoS), which further compresses the model size to 1/3.7 of the original while maintaining comparable model quality.

Beyond architectural innovations, DeepSpeed-MoE also developed a highly optimized inference system to solve deployment challenges for large-scale MoE models. Since MoE models are much larger than equivalent dense models, inference processes are often limited by memory bandwidth rather than compute capacity. By combining model compression with system-level optimizations, this system achieves significant inference acceleration and cost reduction. Specifically, compared to existing MoE inference solutions, DeepSpeed-MoE provides up to 7.3x improvements in latency and cost; compared to dense models of equivalent quality, its inference speed is improved by 4.5x and costs are reduced by 9x. This enables ultra-low-latency inference below 25 milliseconds for trillion-parameter-level MoE models, providing a feasible engineering path for transitioning from dense models to sparse MoE models.