AWS launches managed cluster extensions for SageMaker HyperPod Slurm
AWS introduces declarative, API-driven management of observability and user software on Slurm nodes to prevent configuration drift in AI clusters.
Amazon Web Services (AWS) has launched managed cluster extensions for Amazon SageMaker HyperPod Slurm clusters. This new feature brings the same declarative, API- and console-driven approach used for infrastructure configuration to the software running on the cluster's nodes, specifically targeting observability and user management.
Previously, managing software on shared Slurm clusters relied heavily on lifecycle scripts that ran only during node provisioning. As researchers joined or left, or as teams requested specific metrics, these scripts required manual editing and node replacement, leading to configuration drift where nodes became inconsistent over time. The new extensions allow administrators to declare desired software states via an API or the AWS console. AWS validates these settings before application and ensures consistency across all nodes, including those added by scaling or replacement.
Two initial extensions are available: sagemaker-observability, which collects node, GPU, Elastic Fabric Adapter (EFA), and Slurm job metrics to send to Amazon CloudWatch or Amazon Managed Service for Prometheus; and sagemaker-user-management, which creates users and groups with consistent IDs across nodes, controls SSH/sudo access, registers users with Slurm accounting, and can integrate with Active Directory. Users retain full direct access to nodes for custom configurations, but the extensions provide a managed path for common operational tasks.