AWS Announces NVIDIA Dynamo Support in SageMaker HyperPod Inference
AWS integrates NVIDIA Dynamo into SageMaker HyperPod, offering managed multi-node deployment, disaggregated prefill/decode, and KV-aware routing, with official benchmarks showing over 2x throughput improvement.
Amazon Web Services (AWS) has announced that Amazon SageMaker HyperPod Inference now supports NVIDIA Dynamo. This integration aims to address bottlenecks in efficient Large Language Model (LLM) serving, such as GPU memory coordination and long prompt processing, by combining Dynamo's distributed inference capabilities with HyperPod's infrastructure management to simplify multi-node deployments.
NVIDIA Dynamo is an open-source distributed inference framework supporting multi-node deployment, disaggregated prefill and decode, and KV-aware routing for engines like vLLM, SGLang, and TensorRT-LLM. Within HyperPod, the integration is implemented through custom resource definitions and operators, including the HyperPod Inference Operator, Dynamo operator, Grove (for coordinating pod groups), and the Kubernetes AI (KAI) Scheduler (for GPU resource scheduling).
Users can configure deployments via two paths: "Bring-your-own DynamoGraphDeployment (DGD)" for teams wanting direct control over the serving graph or reusing existing DGDs; and "InferenceEndpointConfig (IEC) with dynamoSpec" for teams deploying new endpoints using a unified configuration format. The latter allows the operator to automatically manage model weight loading, endpoint registration, and autoscaling.
AWS provided benchmark results based on the Qwen3-235B-A22B-FP8 model. On two p5en.48xlarge instances (16 H200 GPUs total) using the TensorRT-LLM engine, disaggregated serving achieved a 2.05x improvement in highest throughput compared to aggregated serving (increasing output tokens per second per GPU from 101 to 207), while meeting performance targets of P90 time-to-first-token under one second and median generation speed of at least 50 tokens per second per user.