NVIDIA Engineers Explain NCCL Symmetric Memory and GPU-Initiated Networking
NVIDIA engineers Jeff Hammond, Hanyue He, and Ke Wen published a tutorial on the PyTorch official channel introducing new NCCL features: symmetric memory and GPU-initiated networking (GIN), and their application in MoE implementations.
ImportanceLocalEvidenceE2 unreplicatedWrite-upQuick
Three NVIDIA engineers, Jeff Hammond, Hanyue He, and Ke Wen, published a tutorial on advanced NCCL features on the official PyTorch channel. The video, released on September 24, 2026, primarily explains two newly introduced functions in NCCL: symmetric memory and GPU-initiated networking (GIN).
According to the publisher's description, these new features are used in state-of-the-art Mixture-of-Experts (MoE) implementations such as DeepEP v2. The tutorial assumes viewers already know how to initialize NCCL and perform common host-initiated operations. It covers how to scale communication within an NVLink domain using symmetric memory and how to scale across interconnected GPU nodes using GIN. All software materials are open source, and content is available in both English and Chinese.