LectureRechercheRadarCadre d'investissement
Connexion / Inscription
← Retour à la lecture
ActualitésSujet · Strata推理引擎Matériel2 min de lectureRédaction DisconfirmAI

How Strata Runs a 180-Billion-Parameter Model on a Gaming PC

The open-source inference engine Strata leverages the Mixture-of-Experts (MoE) architecture of Qwen3.8-Flash-Next to distribute an 180-billion-parameter model across VRAM, system memory, and SSDs, achieving a measured generation speed of approximately 135–145 tokens per second on an RTX 5090. Christian Winkler, author for iX Magazin (Germany), explains the specific mechanisms behind this tiered scheduling.

2026-11-11 14:00 GMT+8

An 180-billion-parameter model sounds impossible to run on a gaming PC. However, the Mixture-of-Experts (MoE) architecture of Qwen3.8-Flash-Next means that only about 6 billion parameters are activated per prediction—selecting 10 out of 512 experts plus one fixed expert. The open-source inference engine Strata is designed around this sparsity with a complete memory scheduling scheme.

Strata’s core idea is not to force the model into VRAM but to place it in tiers: frequently called hot experts remain in GPU VRAM, while other experts are placed in system memory or even on an SSD; the N-Gram embeddings, which account for 51 billion parameters, are loaded on demand only when needed. Combined with 3.5-bit quantization, the weight data volume drops to approximately 55 GB. In tests, the expert cache hit rate on the RTX 5090 platform reached 98%, meaning that the vast majority of inference steps do not require moving data from slow storage.

Tests by author Christian Winkler show that the RTX 5090 paired with Strata can achieve a generation speed of approximately 135–145 tokens/s, with a prefill phase of 2,750 tokens/s; the Mac Studio M2 Ultra with 192 GB of unified memory achieved 49–58 tokens/s. Winkler points out that open-source models have approached the quality of commercial cloud models in certain areas, and running them locally means data stays on your own machine.

Connexion / Inscription
LectureRechercheRadarCadre d'investissement

Sources

  • heise.deiX Magazin↗
Sujet

Strata推理引擎

开源推理引擎 Strata 通过分层调度技术,使 1800 亿参数的 Qwen3.8-Flash-Next 模型能在消费级硬件上运行。该引擎利用 MoE 架构特性,将活跃专家保留在 GPU 显存,非活跃部分卸载至内存和 SSD,实测在 RT…

3 étapes1 piècesMis à jour 2026-11-16 03:25

Comment on en est arrivé là

  1. 2026-10-04Medium 文章介绍 Strata 如何在游戏电脑上运行 125B 参数模型,引发社区关注。medium.com ↗
  2. 2026-11-11德国 iX 杂志发布测试报告,确认 Strata 在 RTX 5090 上实现约 135-145 token/s 的生成速度。heise.de ↗
  3. 2026-11-13GitHub 用户 ExTV 发布针对双卡配置的优化补丁,声称解码速度提升至 250+ token/s。github.com ↗

Ce qui reste non résolu

  • 不同量化格式(如 IQ3_S vs Q2_0)对最终输出质量的具体影响程度如何?
  • Strata 在非 NVIDIA 硬件(如 AMD Radeon, Apple Silicon)上的实际表现是否与官方宣称一致?
  • 随着模型规模进一步增大,现有的 VRAM-RAM-SSD 分层机制是否会遇到新的带宽瓶颈?
Lire le sujet complet →
Continuer la recherche →