How Strata Runs a 180-Billion-Parameter Model on a Gaming PC
The open-source inference engine Strata leverages the Mixture-of-Experts (MoE) architecture of Qwen3.8-Flash-Next to distribute an 180-billion-parameter model across VRAM, system memory, and SSDs, achieving a measured generation speed of approximately 135–145 tokens per second on an RTX 5090. Christian Winkler, author for iX Magazin (Germany), explains the specific mechanisms behind this tiered scheduling.
An 180-billion-parameter model sounds impossible to run on a gaming PC. However, the Mixture-of-Experts (MoE) architecture of Qwen3.8-Flash-Next means that only about 6 billion parameters are activated per prediction—selecting 10 out of 512 experts plus one fixed expert. The open-source inference engine Strata is designed around this sparsity with a complete memory scheduling scheme.
Strata’s core idea is not to force the model into VRAM but to place it in tiers: frequently called hot experts remain in GPU VRAM, while other experts are placed in system memory or even on an SSD; the N-Gram embeddings, which account for 51 billion parameters, are loaded on demand only when needed. Combined with 3.5-bit quantization, the weight data volume drops to approximately 55 GB. In tests, the expert cache hit rate on the RTX 5090 platform reached 98%, meaning that the vast majority of inference steps do not require moving data from slow storage.
Tests by author Christian Winkler show that the RTX 5090 paired with Strata can achieve a generation speed of approximately 135–145 tokens/s, with a prefill phase of 2,750 tokens/s; the Mac Studio M2 Ultra with 192 GB of unified memory achieved 49–58 tokens/s. Winkler points out that open-source models have approached the quality of commercial cloud models in certain areas, and running them locally means data stays on your own machine.