ReadingResearchRadarInvestment framework
Sign in / Sign up
← Back to reading
NewsMaterial2 min readDisconfirmAI Editorial

Community Inference Tools Bring 176-Billion-Parameter Model to Desktop, Achieving ~200 Tokens per Second in Testing

Alibaba's Qwen3.8-Flash-Next utilizes an N-Gram parameter offloading and sparse expert activation architecture, combined with community-developed Strata and Basalt inference engines, enabling a 176-billion-parameter model to achieve approximately 200 tokens per second on a desktop configuration of 128GB RAM and an Nvidia RTX 5090. While quality does not match frontier commercial models, data remains local and there are no usage-based fees.

2026-11-19 17:20 GMT+8

A 176-billion-parameter model running at 200 tokens per second on a desktop PC—this sounds like marketing hype, but Christian Winkler, Professor of NLP at the Nuremberg University of Applied Sciences (TH Nürnberg), verified this figure using his own machine: 128GB RAM plus an Nvidia RTX 5090, running Alibaba's Qwen3.8-Flash-Next via the community inference tool Basalt, a derivative of Strata, actually achieved approximately 200 tokens/s. His assessment: the perceived speed feels faster than some commercial services.

This is possible due to two key layers of model architecture design. Qwen3.8-Flash-Next has a total of 176B parameters, but 51B are so-called N-Gram parameters that can be offloaded to CPU memory or even SSDs, without needing to reside permanently in GPU VRAM. The remaining 122B use a Mixture-of-Experts (MoE) structure, activating only 6B during each inference run. This means only 6 billion parameters are truly involved in computation, similar to a small model, yet the model possesses the knowledge capacity of 122B parameters seen during training. This design decouples 'how large the model is' from 'how much is computed during inference.'

The community tools address the other half of the problem: how to efficiently schedule these parameters scattered across different storage tiers. Strata allows running large models on relatively small graphics cards, provided the system has sufficient memory. It can even run larger models like DeepSeek-V4.1-Flash and GLM-5.3-Flash. Basalt is a fork of Strata, specifically optimized for further improving inference speeds for Qwen3.8-Flash-Next. Winkler noted that these tools largely come from so-called 'vibe coding'—developers using AI assistance for rapid prototyping—and the project maturity and long-term maintainability remain uncertain.

Sign in / Sign up
ReadingResearchRadarInvestment framework

The cost is quality. Winkler explicitly stated that local inference quality does not reach the level of frontier large language models. This represents a trade-off in capability tiers: you gain speed, privacy, and fixed costs, but lose the most complex reasoning abilities and breadth of knowledge. For everyday tasks like writing assistance, code completion, and information summarization, this tier may already be sufficient; for scenarios requiring deep reasoning or precise knowledge retrieval, commercial APIs still lead.

Notable comparisons include other new models mentioned in the same article—Moonshot's Kimi K3.1-Pro has 2.8 trillion parameters, and the so-called Anthropic leaked checkpoint numbat has 14 trillion parameters—these still require datacenter-level hardware. What truly brings large models out of server rooms is not making models smaller, but designing architectures that are 'large in storage, small in computation,' and then using software to execute the separation of storage and computation effectively. Qwen3.8-Flash-Next is currently the model on this path with publicly available weights that can be directly invoked by community tools.

Sources

  • heise.deDeveloper↗
Continue researching →