ReadingResearchRadarInvestment framework
Sign in / Sign up
← Back to reading
NewsMaterial4 min readDisconfirmAI Editorial

Ray Train Breaks Down GPU Starvation into Six Timed Stages to Help Engineers Identify Where to Fix

Anyscale engineer Jason Li introduces new data loading stage timing metrics added in Ray 2.58: each batch passes through six stages—production wait, data transfer, batching, format conversion, collate, and final copy. The metrics show the time share of each stage, helping locate the specific cause of GPU idleness and providing corresponding directions for fixes.

Verified 2026-11-18 02:02 GMT+8

In large-scale model training, fluctuating GPU utilization is the most common signal of performance loss. In the third post of the Ray Train blog series, Anyscale engineer Jason Li points out that a typical cause of this fluctuation is GPU starvation: the training step completes, but the next batch of data is not yet ready, leaving the GPU idle. The problem lies in the fact that there are multiple reasons for data delays—a slow upstream data pipeline, data transmission from other nodes over the network, heavy preprocessing on the training worker's collate_fn, or simply the host-to-device copy taking too long. Each reason requires a completely different fix, yet the symptoms look identical.

Starting with Ray 2.58, Ray Train and Ray Data begin recording how much time each batch spends in each of the six data loading stages. These six stages are sequentially: production wait (waiting for the upstream Ray Data pipeline to produce the next block), data transfer (fetching the block to the training worker, potentially across nodes), batching (slicing and merging blocks into batches, including local shuffle buffers), format conversion (converting to requested formats like NumPy or pandas), collate (executing user-defined per-batch preprocessing on the training worker, such as tokenization or sequence padding), and final copy (moving the batch to the GPU, i.e., host-to-device transfer).

These stages do not execute serially. A background thread pool concurrently prepares the next batch while the current training step runs. As long as the next batch is ready before the training step ends, all loading work is hidden and does not consume wall-clock time. If preparation falls behind, the training loop must wait, causing the GPU to idle. Ray records this wait time (referred to as exposed data loading time) as well as the time spent in each stage on every training card.

Sign in / Sign up
ReadingResearchRadarInvestment framework

The diagnostic path involves three steps. First, check Max Exposed Data Loading Time: if it is zero, data loading keeps up, and tuning the pipeline will not help; if non-zero, this duration represents the GPU idle time per step. Second, examine Percentage Data Loading Breakdown by Stage: the time share of each stage directly points to the bottleneck. A large share of production wait indicates insufficient output from the upstream pipeline; a large share of collate indicates excessive preprocessing on the training worker. Third, split the stage metrics by rank: if one rank is significantly slower than others, it is a straggler. Since all ranks synchronize at every step, a single slow rank can drag down the entire training run.

The article provides specific operational advice for each bottleneck stage. Slow production wait: check the Ray Data dashboard for the slowest operator in the upstream pipeline and tune its parameters, or add CPU-only nodes to allow Ray Data to scale out preprocessing, or use materialize() to cache preprocessed results in memory. Slow data transfer: increase prefetch_batches to start transferring blocks earlier, but be mindful of training node memory usage. Slow batching: reduce local_shuffle_buffer_size or switch to upstream shuffling. Slow format conversion: similarly increase prefetch_batches. Slow collate: move heavy preprocessing from collate_fn to map_batches in the Ray Data pipeline, allowing it to scale across the cluster rather than executing serially on the training worker. Straggling single rank: check if that node has Ray Data tasks competing with the training worker for CPU or memory, or use label_selector to pin the dataset to CPU-only nodes.

It should be noted that this is an official blog post from Anyscale describing its product features. The article does not provide quantitative comparisons before and after using these metrics in real production environments (e.g., how much diagnostic time was shortened or how many percentage points utilization increased). Open-source Ray users need to set up Prometheus and Grafana themselves to view these metrics; the Anyscale platform includes built-in display. For senior teams already familiar with these engineering practices, the advice for each step of the diagnostic path itself is not novel. The new value lies in the stage-level timing metrics, which transform unstable GPU utilization into a problem that can be traced to a single bottleneck, rather than relying on guesswork.

Sources

  • anyscale.comAnyscale↗
Continue researching →