NVIDIA TensorRT Multi-Device Inference: Abstracting Distributed GPU Acceleration into a Single Model Endpoint
An NVIDIA technical blog explains the integration of TensorRT 11.0 and Dynamo-Triton 26.07, allowing a single KIND_MODEL instance to own multiple GPUs and expose one gRPC endpoint. Using Cosmos 3 Nano video generation as an example, it demonstrates reducing latency from 156 seconds to 34 seconds via Ulysses context parallelism, while noting that concurrent throughput and cost efficiency are not covered by this optimization.
ImportanciaMaterialEvidenciaE3 inspeccionableAnálisisRápido
The compute and memory demands of generative AI increasingly exceed what a single GPU can provide. In a technical blog post published on September 21, 2026, NVIDIA detailed how TensorRT Multi-Device Inference integrates with Dynamo-Triton (formerly Triton Inference Server) to simplify model serving across multiple GPUs. Fully supported starting with TensorRT 11.0, this capability enables a single TensorRT network to execute across multiple GPUs using NCCL-backed distributed collectives while retaining TensorRT inference optimizations.
The core change lies in the abstraction of the service interface. In Dynamo-Triton release 26.07, one Triton KIND_MODEL instance can own multiple GPUs, create per-rank TensorRT execution contexts, CUDA streams, and NCCL communicators, and launch the ranks together for each request. Applications call one named model through a gRPC endpoint instead of coordinating GPU ranks themselves. For organizations deploying generative AI, this closes the gap between multi-GPU acceleration and a consumable inference service: teams can trade additional GPU resources for shorter request latency, keep the application interface stable, package the engine as a versioned Triton model, and keep rank and communicator lifecycle code out of the client.
The blog uses NVIDIA Cosmos 3 Nano video generation as a demonstration case. This long-sequence workload features a transformer that accounts for 93.4% of single-GPU generation time, making it the highest-impact stage to accelerate. Diffusers continue to orchestrate prompts, latents, classifier-free guidance (CFG), scheduling, VAE decode, and frame postprocessing, while Dynamo-Triton serves the 36-layer denoising transformer. TensorRT multi-device inference uses Ulysses context parallelism to distribute its 44,160 video tokens across as many as eight NVIDIA GPUs. At context-parallel size eight (CP8), each rank processes 5,520 video tokens outside attention, while within each transformer layer, Ulysses changes the partitioning axis so every rank processes the full video sequence for a nonoverlapping subset of heads.