LesenForschungRadarAnlageframework
Anmelden / Registrieren
← Zurück zur Lektüre
NewsThema · Ray TrainWesentlich4 Min. LesezeitDisconfirmAI Redaktion

Ray Train Breaks Down GPU Starvation into Six Timed Stages to Help Engineers Identify Where to Fix

Anyscale engineer Jason Li introduces new data loading stage timing metrics added in Ray 2.58: each batch passes through six stages—production wait, data transfer, batching, format conversion, collate, and final copy. The metrics show the time share of each stage, helping locate the specific cause of GPU idleness and providing corresponding directions for fixes.

Verifiziert 2026-11-18 02:02 GMT+8

In large-scale model training, fluctuating GPU utilization is the most common signal of performance loss. In the third post of the Ray Train blog series, Anyscale engineer Jason Li points out that a typical cause of this fluctuation is GPU starvation: the training step completes, but the next batch of data is not yet ready, leaving the GPU idle. The problem lies in the fact that there are multiple reasons for data delays—a slow upstream data pipeline, data transmission from other nodes over the network, heavy preprocessing on the training worker's collate_fn, or simply the host-to-device copy taking too long. Each reason requires a completely different fix, yet the symptoms look identical.

Starting with Ray 2.58, Ray Train and Ray Data begin recording how much time each batch spends in each of the six data loading stages. These six stages are sequentially: production wait (waiting for the upstream Ray Data pipeline to produce the next block), data transfer (fetching the block to the training worker, potentially across nodes), batching (slicing and merging blocks into batches, including local shuffle buffers), format conversion (converting to requested formats like NumPy or pandas), collate (executing user-defined per-batch preprocessing on the training worker, such as tokenization or sequence padding), and final copy (moving the batch to the GPU, i.e., host-to-device transfer).

These stages do not execute serially. A background thread pool concurrently prepares the next batch while the current training step runs. As long as the next batch is ready before the training step ends, all loading work is hidden and does not consume wall-clock time. If preparation falls behind, the training loop must wait, causing the GPU to idle. Ray records this wait time (referred to as exposed data loading time) as well as the time spent in each stage on every training card.

Anmelden / Registrieren
LesenForschungRadarAnlageframework

The diagnostic path involves three steps. First, check Max Exposed Data Loading Time: if it is zero, data loading keeps up, and tuning the pipeline will not help; if non-zero, this duration represents the GPU idle time per step. Second, examine Percentage Data Loading Breakdown by Stage: the time share of each stage directly points to the bottleneck. A large share of production wait indicates insufficient output from the upstream pipeline; a large share of collate indicates excessive preprocessing on the training worker. Third, split the stage metrics by rank: if one rank is significantly slower than others, it is a straggler. Since all ranks synchronize at every step, a single slow rank can drag down the entire training run.

The article provides specific operational advice for each bottleneck stage. Slow production wait: check the Ray Data dashboard for the slowest operator in the upstream pipeline and tune its parameters, or add CPU-only nodes to allow Ray Data to scale out preprocessing, or use materialize() to cache preprocessed results in memory. Slow data transfer: increase prefetch_batches to start transferring blocks earlier, but be mindful of training node memory usage. Slow batching: reduce local_shuffle_buffer_size or switch to upstream shuffling. Slow format conversion: similarly increase prefetch_batches. Slow collate: move heavy preprocessing from collate_fn to map_batches in the Ray Data pipeline, allowing it to scale across the cluster rather than executing serially on the training worker. Straggling single rank: check if that node has Ray Data tasks competing with the training worker for CPU or memory, or use label_selector to pin the dataset to CPU-only nodes.

It should be noted that this is an official blog post from Anyscale describing its product features. The article does not provide quantitative comparisons before and after using these metrics in real production environments (e.g., how much diagnostic time was shortened or how many percentage points utilization increased). Open-source Ray users need to set up Prometheus and Grafana themselves to view these metrics; the Anyscale platform includes built-in display. For senior teams already familiar with these engineering practices, the advice for each step of the diagnostic path itself is not novel. The new value lies in the stage-level timing metrics, which transform unstable GPU utilization into a problem that can be traced to a single bottleneck, rather than relying on guesswork.

Quellen

  • anyscale.comAnyscale↗
Thema

Ray Train

Ray Train 在 2.58 版本中引入了数据加载可观测性指标,旨在解决分布式训练中常见的“GPU 饥饿”问题。当 GPU 利用率波动或训练变慢时,往往是因为数据准备跟不上计算速度,但传统监控难以定位具体瓶颈。新指标将每个批次的数据加载…

3 Schritte1 BeiträgeAktualisiert 2026-11-20 11:36

Wie es dazu kam

  1. 2026-06-07GitHub Issue #63911 提出端到端可观测性 RFC,指出 Ray Data 与 Ray Train 监控体系独立,存在中间盲区。github.com ↗
  2. 2026-06-16GitHub Issue #64132 进一步细化数据摄入可观测性方案,提议增加按阶段分解的训练线程阻塞时间指标。github.com ↗
  3. 2026-11-17Anyscale 博客文章《Diagnosing GPU Starvation》发布,介绍 Ray 2.58 中正式推出的六阶段数据加载计时功能。anyscale.com ↗

Was noch ungeklärt ist

  • 这些新指标在超大规模集群(如数千 GPU)中的性能开销是多少?
  • 除了开源 Ray,Anyscale 平台上的 KubeRay Connect 如何具体集成这些实时面板?
  • 对于非 PyTorch 框架(如 TensorFlow/JAX),这六个阶段的映射是否完全一致?
Vollständiges Thema lesen →
Recherche fortsetzen →