NVIDIA Technical Blog: How CUDA Green Contexts Cut Critical Kernel Latency to Microseconds via Explicit Partitioning
NVIDIA engineers explain the Green Contexts mechanism introduced in CUDA 13.1. By explicitly allocating Streaming Multiprocessors (SMs) and workqueues, the mechanism allows applications to isolate latency-sensitive tasks from high-throughput background jobs within a single process. Tests on a Blackwell GPU show that compared to using only high-priority streams, Green Contexts reduce critical kernel wait time from 0.140 ms to 0.007 ms by bypassing the scheduler's need to wait for existing blocks to drain.
ImportanceMaterialEvidenceE3 inspectableWrite-upQuick
Modern GPU applications often run multiple independent components simultaneously within a single process: communication overlap in distributed training, real-time inference in AI sensor platforms, or parallel data preprocessing and model inference. Traditionally, developers relied on CUDA stream priorities to ensure critical tasks weren't blocked, but this approach has a physical ceiling. When high-throughput bulk workloads saturate all Streaming Multiprocessors (SMs), even a highest-priority critical kernel must wait for at least one SM to free up before it can begin execution. This 'drain wait' causes unpredictable latency jitter.
To address this limitation, NVIDIA officially introduced Green Contexts in the CUDA 13.1 Runtime API. Unlike traditional heavy device contexts, Green Contexts are a lightweight programming model that allows applications to explicitly define and partition GPU execution resources. Developers can query device resources, package a specific subset of SMs and workqueue configurations into a descriptor, and create independent execution contexts based on it. Subsequently, application-created streams bind directly to these contexts, clearly specifying which compute units serve which types of tasks.