Anyscale Field Engineer Naila Khan: KubeRay Default Configurations Fail at Five Boundaries in Multi-Team GPU Clusters
When RayJobs scale from a single successful run to hundreds of jobs across multiple teams and clusters, default configurations that worked well at small scale begin to fail under production parallelism and contention. This is the core observation from Part 2 of the KubeRay series. She emphasizes that these are not defects in KubeRay itself, but boundary issues between Ray, Kubernetes, and the organization running both.
She describes two types of teams from her perspective as a field engineer: enterprises fitting Ray into clusters built for other purposes, and digital natives pushing Ray beyond its previous limits. She notes they repeatedly encounter the same five scenarios. The first scenario is debugging data tied to the cluster lifecycle: a RayJob configured with shutdownAfterJobFinishes: true deletes its Ray dashboard while cleaning up after itself. The hardest case to troubleshoot is when the driver process exits within the first few minutes—GPU worker nodes start, receive no tasks, and the cluster is torn down before anyone can capture the driver logs.
The second scenario is hardware faults surfacing as application errors. PCIe or NVLink faults, or hung NFS mounts, can interrupt a training run while the Pod still reports Ready status. Ray continues to schedule work to that node, so the failure presents as an application error. Some teams have built modified operators and custom resources to detect failing hardware.
The third scenario is that default settings require tuning at production scale. Khan gave specific examples: when 20 clusters compete for one GPU pool, idle GPU worker nodes may be reaped while their placement group is still forming, preventing training from starting; setting maxReplicas with no headroom can stall the autoscaler when Ray Train retries out-of-memory (OOM) tasks; each retry can also leave a placement group behind. She stated that field team feedback indicates weeks of platform engineering work are needed before reliability stabilizes.
The fourth scenario is head node disruptions affecting the entire cluster. Khan cited the experience of the Klaviyo engineering team, which has been running Ray Serve on KubeRay for a year: node drains for node-image updates and security patches can take down Ray nodes, and memory pressure can evict the head Pod. Without GCS fault tolerance, the whole cluster fails; with it enabled, workers continue running only if the head recovers within about ten minutes, which is KubeRay's default worker reconnect timeout.
The fifth scenario is coexistence of Ray with existing platform services. Service meshes, shared namespaces, quotas, and audit requirements predate Ray. Khan described an enterprise team serving thousands of models on Ray Serve that found Envoy sidecars creating a listener for every Ray port, which strained the mesh control plane and affected neighboring services.
For these five scenarios, Khan offers six practices, emphasizing that none require new tools but instead leverage capabilities already available in the Kubernetes ecosystem. Core recommendations include: enabling the History Server (introduced as alpha in KubeRay v1.6, promoted to beta in v1.7) to write events and logs to object storage; running the NVIDIA DCGM exporter and alerting on GPU Xid errors rather than just monitoring utilization; using a gang scheduler to ensure job worker nodes start together or not at all; leaving headroom of at least two additional worker nodes for to accommodate replacements after OOM retries; setting on the head node; and keeping the service mesh at the edge while excluding Ray's internal ports.
Quellen:Anyscale