NVIDIA Solutions Architect Yang Liu: Missing Interface in DGX Spark MoE Inference Optimization Causes Performance Below Baseline
Yang Liu documented the process of applying three Mixture-of-Experts (MoE) inference optimizations to the Ling-3.0-flash model on NVIDIA DGX Spark in a blog post for NVIDIA developers. The core finding is that while Humming kernel replacement, online FP8 LM head quantization, and NEXTN speculative decoding are each effective individually, enabling both FP8 LM head quantization and NEXTN speculative decoding simultaneously requires the draft model to share the complete LM head module; otherwise, the framework silently falls back, resulting in throughput lower than the unoptimized baseline. Additionally, the 185.1% throughput gain measured on synthetic workloads was only 42.5% on the real GSM8K evaluation, reminding developers to verify optimization effects using actual deployment workloads.
In edge inference scenarios with a batch size of 1, the performance bottleneck for Mixture-of-Experts (MoE) models is often not compute power but memory bandwidth. NVIDIA Solutions Architect Yang Liu recorded this assessment in a developer blog post dated October 27, 2026: profiling of Inclusion AI’s Ling-3.0-flash—a MoE model with 124 billion total parameters and 5.1 billion active parameters per token—on NVIDIA DGX Spark showed that the decoding phase was limited by weight reads, with two hotspot operations being MoE expert routing computation and output-layer vocabulary projection.
To address these two bottlenecks, Yang Liu tested three complementary optimizations: replacing the FlashInfer MXFP4 MoE backend with the Humming kernel library (leveraging NVRTC runtime compilation to adapt to specific dimensions, yielding a 9.3% throughput increase); applying online FP8 quantization to the LM head to halve its weight read volume (increasing throughput by 21.7% when combined with Humming); and enabling the model’s built-in NEXTN layer as a draft model for multi-token speculative decoding (achieving a 185.1% throughput increase under synthetic workloads). These three mechanisms do not overlap—the first two reduce the cost per step, while the third reduces the number of steps—and can theoretically be stacked.