LeituraPesquisaRadarFramework de investimento
Entrar / Cadastrar
← Voltar para a leitura
NotíciasTópico · SlipStreamMaterial2 min de leituraEditorial DisconfirmAI

SlipStream: Demand-Driven Prefetching Significantly Mitigates High-Bandwidth Flash GPU Inference Latency

A team from Georgia Tech, NVIDIA Research, and Stanford University has proposed SlipStream, a software-hardware co-designed prefetcher that transforms the prefetching of flash weights in LLM inference into a demand-driven pipeline. Using a 40 MB buffer, it achieves 92.4% of ideal HBF performance, whereas the previous H3 solution reached only 43.1% with a 320 MB buffer.

2026-11-17 07:52 GMT+8

High-Bandwidth Flash (HBF) replaces DRAM chips in HBM with NAND flash, offering 16 times the capacity at comparable read bandwidths, and is viewed as a key path for expanding LLM inference memory. However, HBF's single-read latency approaches 20 microseconds—20 times that of HBM—and on-demand access to HBF can slow inference speeds by up to 12.8 times. A research team from Georgia Tech, NVIDIA Research, and Stanford University proposed SlipStream in an arXiv preprint, attempting to significantly mitigate this latency penalty through software-hardware co-design.

The core idea of SlipStream is to turn prefetching into a pipeline controlled by demand rhythm. On the software side, it first analyzes the repeatable weight-page access order in data-independent LLM kernels, then inserts a staging warp that prefetches pages in that order using non-blocking STAGE instructions. On the hardware side, the staging warp acts as the producer, while work warps act as consumers. The key design decision is that each STAGE tracking entry is not retired when the page returns from HBF, but rather when the work warp first accesses that page on demand. This means the demand rhythm naturally limits the distance between prefetching leading consumption, thereby decoupling buffer size from in-flight traffic.

Across seven dense models spanning 16 to 800 GB, SlipStream uses a 40 MB staging buffer to achieve 92.4% of the performance of ideal HBF (assuming HBM-level latency). By comparison, the H3 tensor-streaming scheme reaches only 43.1% even with a 320 MB buffer. SlipStream's buffer size is independent of HBF latency and workload: a 16 MB buffer performs within 4 percentage points of the 40 MB version, while the 16 MB buffer itself is 20 times smaller than the 320 MB buffer required by H3. The solution also supports Mixture-of-Experts models, achieving 91.2% of ideal performance with a 40 MB buffer.

Entrar / Cadastrar
LeituraPesquisaRadarFramework de investimento

It should be noted that this is an arXiv preprint from November 2026; all performance figures come from self-tests by the author team, with no independent reproduction yet available. SlipStream relies on offline profiling of kernel weight access orders, predicated on the assumption that kernels are data-independent—meaning access patterns do not change with input data. For dynamic workloads with unpredictable access orders, the effectiveness of this mechanism remains to be verified.

Fontes

  • semiengineering.com↗
Tópico

SlipStream

【背景】高带宽闪存(HBF)虽能提供比HBM高16倍的容量,但其约20微秒的读取延迟是GPU推理的主要瓶颈。Georgia Tech、NVIDIA Research和斯坦福大学团队提出SlipStream,一种软硬件协同设计的预取器,将LL…

2 etapas1 peçasAtualizado 2026-11-19 19:34

Como chegamos aqui

  1. 2026-11-10论文《SlipStream: Enabling HBF for GPU Systems by Mitigating the Latency and SRAM Overheads》提交至arXiv。arxiv.org ↗
  2. 2026-11-17SemiEngineering等媒体报道SlipStream技术,指出其以40MB缓冲实现92.4%理想性能,远超H3方案。semiengineering.com ↗

O que ainda está indefinido

  • SlipStream在实际物理GPU硬件上的部署效果是否与模拟结果一致?
  • 该技术是否已被纳入NVIDIA或其他厂商的下一代GPU架构路线图?
Ler o tópico completo →
Continuar pesquisando →