読書リサーチレーダー投資フレームワーク
ログイン / 新規登録
← 読解に戻る
ニューストピック · KubeRay重大4 分で読めるDisconfirmAI 編集部

Anyscale Field Engineer Naila Khan: KubeRay Default Configurations Fail at Five Boundaries in Multi-Team GPU Clusters

Based on field engineering experience, Naila Khan describes five specific failure scenarios that Ray encounters when expanding from a single-team single-cluster setup to multi-team shared GPU pools: debugging data disappearing with the cluster, hardware faults masquerading as application errors, autoscaling defaults mismatching under contention, head node interruptions affecting the entire cluster, and conflicts between service meshes and Ray's internal ports. She provides corresponding best practices for each.

検証済み 2026-10-28 06:59 GMT+8

When RayJobs scale from a single successful run to hundreds of jobs across multiple teams and clusters, default configurations that worked well at small scale begin to fail under production parallelism and contention. This is the core observation from Part 2 of the KubeRay series. She emphasizes that these are not defects in KubeRay itself, but boundary issues between Ray, Kubernetes, and the organization running both.

She describes two types of teams from her perspective as a field engineer: enterprises fitting Ray into clusters built for other purposes, and digital natives pushing Ray beyond its previous limits. She notes they repeatedly encounter the same five scenarios. The first scenario is debugging data tied to the cluster lifecycle: a RayJob configured with shutdownAfterJobFinishes: true deletes its Ray dashboard while cleaning up after itself. The hardest case to troubleshoot is when the driver process exits within the first few minutes—GPU worker nodes start, receive no tasks, and the cluster is torn down before anyone can capture the driver logs.

The second scenario is hardware faults surfacing as application errors. PCIe or NVLink faults, or hung NFS mounts, can interrupt a training run while the Pod still reports Ready status. Ray continues to schedule work to that node, so the failure presents as an application error. Some teams have built modified operators and custom resources to detect failing hardware.

ログイン / 新規登録
読書リサーチレーダー投資フレームワーク

The third scenario is that default settings require tuning at production scale. Khan gave specific examples: when 20 clusters compete for one GPU pool, idle GPU worker nodes may be reaped while their placement group is still forming, preventing training from starting; setting maxReplicas with no headroom can stall the autoscaler when Ray Train retries out-of-memory (OOM) tasks; each retry can also leave a placement group behind. She stated that field team feedback indicates weeks of platform engineering work are needed before reliability stabilizes.

The fourth scenario is head node disruptions affecting the entire cluster. Khan cited the experience of the Klaviyo engineering team, which has been running Ray Serve on KubeRay for a year: node drains for node-image updates and security patches can take down Ray nodes, and memory pressure can evict the head Pod. Without GCS fault tolerance, the whole cluster fails; with it enabled, workers continue running only if the head recovers within about ten minutes, which is KubeRay's default worker reconnect timeout.

The fifth scenario is coexistence of Ray with existing platform services. Service meshes, shared namespaces, quotas, and audit requirements predate Ray. Khan described an enterprise team serving thousands of models on Ray Serve that found Envoy sidecars creating a listener for every Ray port, which strained the mesh control plane and affected neighboring services.

For these five scenarios, Khan offers six practices, emphasizing that none require new tools but instead leverage capabilities already available in the Kubernetes ecosystem. Core recommendations include: enabling the History Server (introduced as alpha in KubeRay v1.6, promoted to beta in v1.7) to write events and logs to object storage; running the NVIDIA DCGM exporter and alerting on GPU Xid errors rather than just monitoring utilization; using a gang scheduler to ensure job worker nodes start together or not at all; leaving headroom of at least two additional worker nodes for maxReplicas to accommodate replacements after OOM retries; setting num-cpus: "0" on the head node; and keeping the service mesh at the edge while excluding Ray's internal ports.

At the end of the article, Khan proposed a decision framework: if you need to maintain a dedicated pipeline to capture logs before RayJob termination, spend hours troubleshooting whether a failure was due to the GPU, Kubernetes, or Ray, manually tune autoscaling defaults, run redundant clusters or custom health checkers, manually map workloads to teams for auditing or chargeback, or operate Ray across multiple clusters, clouds, and regions—if more than two of these describe your week, a platform layer may be worth considering. She then introduced Anyscale’s KubeRay Connect private preview as a concrete implementation of this layer.

It should be noted that this article is published by Anyscale, whose commercial product KubeRay Connect is precisely the solution recommended in the article. The six practices themselves are based on verifiable Kubernetes and Ray ecosystem tools, but the criteria for 'when to introduce a platform layer' align with Anyscale's commercial interests. Readers can treat the practices section as independent technical reference material, while viewing the product recommendation as a vendor claim.

Khan also mentioned a notable detail: seeing logs and dashboards for completed RayJobs was one of the earliest requests in the KubeRay community, with users raising it as early as 2023. The difficulty arises because ephemeral clusters are the very design intent of RayJob, meaning evidence must leave the cluster before the cluster disappears. The History Server is the community's response, built with contributors from several large platform teams.

ソース

  • anyscale.comAnyscale↗
トピック

KubeRay

KubeRay作为连接Ray与Kubernetes的关键组件,其默认配置在单团队小规模测试中表现良好,但在扩展至多团队共享GPU池的生产环境时,往往在调试数据留存、硬件故障识别、自动扩缩容争用、头节点高可用及服务网格兼容性这五个边界处失效。

3 ステップ1 件更新日 2026-10-28 23:02

経緯

  1. 2026-10-26Anyscale发布博客《KubeRay in production: best practices from the field》,由现场工程师Naila Khan撰写,详细描述了KubeRay在多团队GPU集群中的五个典型失效场景及六项最佳实践。anyscale.com ↗
  2. 2025-07-10Ray项目合并PR #54229,新增《Troubleshooting DeepSeek/multi-node GPU deployment on KubeRay》文档,提供针对LLM多节点GPU服务的系统性调试策略,包括区分单节点与KubeRay行为、变体硬件配置测试等。github.com ↗
  3. 2024-11-27GitHub Issue #30648报告Bitnami KubeRay Helm Chart (v1.2.19)因缺失Endpoints资源的RBAC规则,导致RayService状态卡在`WaitForServeDeploymentReady`,凸显了权限配置对服务启动的影响。github.com ↗

未解決の問題

  • KubeRay History Server从alpha到beta的演进是否已完全解决临时集群日志丢失问题,特别是在驱动极早崩溃的场景下?
  • 针对H100等新型GPU实例的NCCL拓扑检测失败,是否有标准化的AWS OFI插件修复方案被纳入KubeRay默认镜像?
  • 在多租户环境下,如何平衡Kueue配额管理与Ray内部自动扩缩容机制,以避免placement group形成期间的资源争用?
トピック全文を読む →
リサーチを続ける →