Ray Train Breaks Down GPU Starvation into Six Timed Stages to Help Engineers Identify Where to Fix
Anyscale engineer Jason Li introduces new data loading stage timing metrics added in Ray 2.58: each batch passes through six stages—production wait, data transfer, batching, format conversion, collate, and final copy. The metrics show the time share of each stage, helping locate the specific cause of GPU idleness and providing corresponding directions for fixes.
In large-scale model training, fluctuating GPU utilization is the most common signal of performance loss. In the third post of the Ray Train blog series, Anyscale engineer Jason Li points out that a typical cause of this fluctuation is GPU starvation: the training step completes, but the next batch of data is not yet ready, leaving the GPU idle. The problem lies in the fact that there are multiple reasons for data delays—a slow upstream data pipeline, data transmission from other nodes over the network, heavy preprocessing on the training worker's collate_fn, or simply the host-to-device copy taking too long. Each reason requires a completely different fix, yet the symptoms look identical.
Starting with Ray 2.58, Ray Train and Ray Data begin recording how much time each batch spends in each of the six data loading stages. These six stages are sequentially: production wait (waiting for the upstream Ray Data pipeline to produce the next block), data transfer (fetching the block to the training worker, potentially across nodes), batching (slicing and merging blocks into batches, including local shuffle buffers), format conversion (converting to requested formats like NumPy or pandas), collate (executing user-defined per-batch preprocessing on the training worker, such as tokenization or sequence padding), and final copy (moving the batch to the GPU, i.e., host-to-device transfer).
These stages do not execute serially. A background thread pool concurrently prepares the next batch while the current training step runs. As long as the next batch is ready before the training step ends, all loading work is hidden and does not consume wall-clock time. If preparation falls behind, the training loop must wait, causing the GPU to idle. Ray records this wait time (referred to as exposed data loading time) as well as the time spent in each stage on every training card.