Anyscale Field Engineer Naila Khan: KubeRay Default Configurations Fail at Five Boundaries in Multi-Team GPU Clusters
Based on field engineering experience, Naila Khan describes five specific failure scenarios that Ray encounters when expanding from a single-team single-cluster setup to multi-team shared GPU pools: debugging data disappearing with the cluster, hardware faults masquerading as application errors, autoscaling defaults mismatching under contention, head node interruptions affecting the entire cluster, and conflicts between service meshes and Ray's internal ports. She provides corresponding best practices for each.
When RayJobs scale from a single successful run to hundreds of jobs across multiple teams and clusters, default configurations that worked well at small scale begin to fail under production parallelism and contention. This is the core observation from Part 2 of the KubeRay series. She emphasizes that these are not defects in KubeRay itself, but boundary issues between Ray, Kubernetes, and the organization running both.
She describes two types of teams from her perspective as a field engineer: enterprises fitting Ray into clusters built for other purposes, and digital natives pushing Ray beyond its previous limits. She notes they repeatedly encounter the same five scenarios. The first scenario is debugging data tied to the cluster lifecycle: a RayJob configured with shutdownAfterJobFinishes: true deletes its Ray dashboard while cleaning up after itself. The hardest case to troubleshoot is when the driver process exits within the first few minutes—GPU worker nodes start, receive no tasks, and the cluster is torn down before anyone can capture the driver logs.
The second scenario is hardware faults surfacing as application errors. PCIe or NVLink faults, or hung NFS mounts, can interrupt a training run while the Pod still reports Ready status. Ray continues to schedule work to that node, so the failure presents as an application error. Some teams have built modified operators and custom resources to detect failing hardware.
The third scenario is that default settings require tuning at production scale. Khan gave specific examples: when 20 clusters compete for one GPU pool, idle GPU worker nodes may be reaped while their placement group is still forming, preventing training from starting; setting with no headroom can stall the autoscaler when Ray Train retries out-of-memory (OOM) tasks; each retry can also leave a placement group behind. She stated that field team feedback indicates weeks of platform engineering work are needed before reliability stabilizes.