ReadingResearchRadarInvestment framework
Sign in / Sign up
← Back to reading
NewsMaterial4 min readDisconfirmAI Editorial

Hugging Face Blog Author Tarek Ziadé Runs 180B MoE Model on Transformers with 32GB Mac, Identifies Host Round-Trip as Remaining Bottleneck

Hugging Face blog author Tarek Ziadé implemented streaming loading of Mixture-of-Experts (MoE) experts from SSD in Transformers, running the 180B-parameter Qwen3.8-Flash-Next model on a 32GB MacBook M5 at approximately 94% of the decoding speed of the dedicated engine Strata. His experiments indicate that the remaining performance gap is primarily due to host round-trip overhead of about 10 milliseconds per token, rather than disk read latency for expert weights.

2026-11-18 08:00 GMT+8

Tarek Ziadé has been working to reduce reliance on online models by running larger ones locally. His workstation is a 32GB unified-memory MacBook M5, with the goal of running the 180B-parameter Qwen3.8-Flash-Next model on this laptop—a sparse MoE model with 48 layers, each containing 512 experts (with 10 activated per token). Expert weights account for over 90% of the model's total parameters; in FP8 format, the experts alone total 121GB, far exceeding the device's memory capacity.

The community previously recommended using Strata, an inference engine designed specifically for MoE models, which keeps only 'hot' experts in VRAM while storing the rest in RAM or even on disk. On this machine, Strata achieved a decoding speed of approximately 14.8 tok/s. Ziadé's question was: could he achieve comparable performance within the general-purpose Transformers framework without introducing a specialized engine?

The key to the answer lies in a mechanism he calls the 'slot pool.' The initial version stored experts on disk and managed them with an LRU cache, loading one expert at a time and computing it via a Python loop—the result was 10.1 tok/s, significantly lower than the native Transformers speed of roughly 32 tok/s. The issue was not I/O but the loop: the native implementation uses grouped_mm to run all experts in a layer in a single kernel call.

The slot pool approach involves preallocating a device tensor shared across all layers, along with a per-layer mapping table from expert ID to slot. After the router selects the experts needed for the current token, their weights are loaded into the corresponding slots, and the router's expert IDs are remapped to slot indices. This makes the slot pool appear to as a standard set of expert weights, requiring no custom matrix multiplication. This change increased the speed on OLMoE-1B-7B from 10.1 to 34.6 tok/s (compared to 38.1 for native).

Sign in / Sign up
ReadingResearchRadarInvestment framework
grouped_mm

Porting this mechanism to Qwen3.8-Flash-Next required addressing an additional issue: the model includes a 51GB n-gram embedding table that also had to be streamed from disk. The initial configuration quantized experts to 3 bits for disk storage (49GB), with a 12GiB device cache; dense layers were quantized to 8 bits (approximately 5.4GB); and the n-gram table was read row-by-row on demand. After loading, usage was approximately 18GiB, peaking at 25.4GiB for a 6K-token prompt—approaching the 32GB limit.

After multiple rounds of optimization—including early GPU work submission, Metal kernels, switching to Strata's GGUF weights to reduce dense weight traffic per token from 5.4GB to 3.9GB, prefetching the next layer's experts, and improving long-document attention—the final decode speed reached 13.9 tok/s on 200 tokens, compared to Strata's 14.8 tok/s, a gap of about 6%. Prefill speed was slightly faster than Strata's.

However, Ziadé identified the true remaining bottleneck in his conclusion: most of the remaining time was spent on host round-trips, averaging about 10 milliseconds per token. Synchronous wake-up accounted for only about 1 millisecond of that; the rest was the GPU idling while Python queued work for the next layer. Strata queues the entire token's work upfront, allowing the GPU to wait on a Metal shared event at each layer while the host reads experts. PyTorch does not expose this fine-grained control. He attempted to simulate it with a spinning kernel watching for CPU writes, but the GPU sometimes took a second or never saw the write. A real fix would require native operations on PyTorch's MPS stream, carrying a risk of deadlocks, which he did not implement.

Ziadé's conclusion was honest: ~14 tok/s is still too slow, and his primary coding work remains on his home desktop equipped with dual RTX 4090s. However, he believes the value of this experiment lies in demonstrating how a general-purpose framework can approach the performance of a dedicated engine: by streaming only routed experts, using a bounded cache compatible with grouped_mm, and employing a few targeted Metal kernels. Closing the final 6% gap requires deeper framework changes, starting with per-layer synchronous control, which PyTorch currently does not provide.

Sources

  • huggingface.co↗
Continue researching →