Community Inference Tools Bring 176-Billion-Parameter Model to Desktop, Achieving ~200 Tokens per Second in Testing
Alibaba's Qwen3.8-Flash-Next utilizes an N-Gram parameter offloading and sparse expert activation architecture, combined with community-developed Strata and Basalt inference engines, enabling a 176-billion-parameter model to achieve approximately 200 tokens per second on a desktop configuration of 128GB RAM and an Nvidia RTX 5090. While quality does not match frontier commercial models, data remains local and there are no usage-based fees.
A 176-billion-parameter model running at 200 tokens per second on a desktop PC—this sounds like marketing hype, but Christian Winkler, Professor of NLP at the Nuremberg University of Applied Sciences (TH Nürnberg), verified this figure using his own machine: 128GB RAM plus an Nvidia RTX 5090, running Alibaba's Qwen3.8-Flash-Next via the community inference tool Basalt, a derivative of Strata, actually achieved approximately 200 tokens/s. His assessment: the perceived speed feels faster than some commercial services.
This is possible due to two key layers of model architecture design. Qwen3.8-Flash-Next has a total of 176B parameters, but 51B are so-called N-Gram parameters that can be offloaded to CPU memory or even SSDs, without needing to reside permanently in GPU VRAM. The remaining 122B use a Mixture-of-Experts (MoE) structure, activating only 6B during each inference run. This means only 6 billion parameters are truly involved in computation, similar to a small model, yet the model possesses the knowledge capacity of 122B parameters seen during training. This design decouples 'how large the model is' from 'how much is computed during inference.'
The community tools address the other half of the problem: how to efficiently schedule these parameters scattered across different storage tiers. Strata allows running large models on relatively small graphics cards, provided the system has sufficient memory. It can even run larger models like DeepSeek-V4.1-Flash and GLM-5.3-Flash. Basalt is a fork of Strata, specifically optimized for further improving inference speeds for Qwen3.8-Flash-Next. Winkler noted that these tools largely come from so-called 'vibe coding'—developers using AI assistance for rapid prototyping—and the project maturity and long-term maintainability remain uncertain.