SlipStream: Demand-Driven Prefetching Significantly Mitigates High-Bandwidth Flash GPU Inference Latency
A team from Georgia Tech, NVIDIA Research, and Stanford University has proposed SlipStream, a software-hardware co-designed prefetcher that transforms the prefetching of flash weights in LLM inference into a demand-driven pipeline. Using a 40 MB buffer, it achieves 92.4% of ideal HBF performance, whereas the previous H3 solution reached only 43.1% with a 320 MB buffer.
High-Bandwidth Flash (HBF) replaces DRAM chips in HBM with NAND flash, offering 16 times the capacity at comparable read bandwidths, and is viewed as a key path for expanding LLM inference memory. However, HBF's single-read latency approaches 20 microseconds—20 times that of HBM—and on-demand access to HBF can slow inference speeds by up to 12.8 times. A research team from Georgia Tech, NVIDIA Research, and Stanford University proposed SlipStream in an arXiv preprint, attempting to significantly mitigate this latency penalty through software-hardware co-design.
The core idea of SlipStream is to turn prefetching into a pipeline controlled by demand rhythm. On the software side, it first analyzes the repeatable weight-page access order in data-independent LLM kernels, then inserts a staging warp that prefetches pages in that order using non-blocking STAGE instructions. On the hardware side, the staging warp acts as the producer, while work warps act as consumers. The key design decision is that each STAGE tracking entry is not retired when the page returns from HBF, but rather when the work warp first accesses that page on demand. This means the demand rhythm naturally limits the distance between prefetching leading consumption, thereby decoupling buffer size from in-flight traffic.
Across seven dense models spanning 16 to 800 GB, SlipStream uses a 40 MB staging buffer to achieve 92.4% of the performance of ideal HBF (assuming HBM-level latency). By comparison, the H3 tensor-streaming scheme reaches only 43.1% even with a 320 MB buffer. SlipStream's buffer size is independent of HBF latency and workload: a 16 MB buffer performs within 4 percentage points of the 40 MB version, while the 16 MB buffer itself is 20 times smaller than the 320 MB buffer required by H3. The solution also supports Mixture-of-Experts models, achieving 91.2% of ideal performance with a 40 MB buffer.