Hugging Face Blog Author Tarek Ziadé Runs 180B MoE Model on Transformers with 32GB Mac, Identifies Host Round-Trip as Remaining Bottleneck
Hugging Face blog author Tarek Ziadé implemented streaming loading of Mixture-of-Experts (MoE) experts from SSD in Transformers, running the 180B-parameter Qwen3.8-Flash-Next model on a 32GB MacBook M5 at approximately 94% of the decoding speed of the dedicated engine Strata. His experiments indicate that the remaining performance gap is primarily due to host round-trip overhead of about 10 milliseconds per token, rather than disk read latency for expert weights.
Tarek Ziadé has been working to reduce reliance on online models by running larger ones locally. His workstation is a 32GB unified-memory MacBook M5, with the goal of running the 180B-parameter Qwen3.8-Flash-Next model on this laptop—a sparse MoE model with 48 layers, each containing 512 experts (with 10 activated per token). Expert weights account for over 90% of the model's total parameters; in FP8 format, the experts alone total 121GB, far exceeding the device's memory capacity.
The community previously recommended using Strata, an inference engine designed specifically for MoE models, which keeps only 'hot' experts in VRAM while storing the rest in RAM or even on disk. On this machine, Strata achieved a decoding speed of approximately 14.8 tok/s. Ziadé's question was: could he achieve comparable performance within the general-purpose Transformers framework without introducing a specialized engine?
The key to the answer lies in a mechanism he calls the 'slot pool.' The initial version stored experts on disk and managed them with an LRU cache, loading one expert at a time and computing it via a Python loop—the result was 10.1 tok/s, significantly lower than the native Transformers speed of roughly 32 tok/s. The issue was not I/O but the loop: the native implementation uses grouped_mm to run all experts in a layer in a single kernel call.
The slot pool approach involves preallocating a device tensor shared across all layers, along with a per-layer mapping table from expert ID to slot. After the router selects the experts needed for the current token, their weights are loaded into the corresponding slots, and the router's expert IDs are remapped to slot indices. This makes the slot pool appear to as a standard set of expert weights, requiring no custom matrix multiplication. This change increased the speed on OLMoE-1B-7B from 10.1 to 34.6 tok/s (compared to 38.1 for native).