TensorRT 11.1 Auto-Tuning: Turning Compiler Scheduling into a Searchable Build Route
The NVIDIA TensorRT team released the Global Performance Tuner in version 11.1, whose core design represents compiler scheduling strategies as explicit build routes composed of over 200 knob combinations. Developers set accuracy tolerance as a constraint; the tuner iteratively builds, measures, and discards ineligible candidates within the route space to select the fastest compliant engine. The authors use a thought experiment on quantization insertion to illustrate how performance and accuracy shift from entangled to separable optimization, noting that optimal routes are strictly bound to specific models, shapes, hardware, and software stacks, preventing direct reuse across environments.
When generating inference engines, the scheduling stage of deep learning compilers determines the order and manner in which layer operations are executed. Traditionally, compiler authors wrote heuristic rules for common models and added exceptions when encountering new models or hardware combinations. In a technical blog post published on October 23 by Jian Hu, Senior System Software Engineer at NVIDIA TensorRT, and Lin Chai, Senior Product Manager at NVIDIA TensorRT, they describe how they changed this workflow: representing scheduling strategies as searchable build routes, shifting performance optimization from prediction to measurement.
The core concept of a build route is the compiler knob. A knob is a configuration item that changes an aspect of the compilation pipeline, such as choosing between horizontal and vertical fusion or selecting a code generation path. TensorRT 11.1 exposes more than 200 tunable knobs, each with a finite set of allowed values and a default value. A determined combination of knob values constitutes a build route. When no route is specified, TensorRT uses the default build route where all knobs retain their default values.
The authors use a hypothetical three-pass compiler to explain the basic logic: the fusion strategy of the second pass, f₂, is controlled by a binary knob p. When p=1, horizontal fusion is applied; when p=0, vertical fusion is applied. These two values produce two routes, generating two candidate engines for the same input model. Horizontal fusion is not always faster—it may win for one model but lose for another, and results also vary with tensor shapes and target GPUs. Therefore, selecting the better route requires building both candidates and comparing their end-to-end performance.
This two-route example captures the essence of tuning: the compiler exposes meaningful scheduling alternatives, and measured behavior determines which complete solution performs best in the current deployment scenario. Hu and Chai emphasize that knob combinations are specifications for build routes, not complete descriptions of the engine; TensorRT still executes remaining construction stages such as tactic selection and may reconcile individual knob values with network constraints.
In practice, the trtexec tool organizes tuning as a loop: users specify an input model and a regex-like route expression. trtexec expands the expression into a list of concrete build routes according to a search algorithm, builds engines for each route, and measures accuracy and performance. Routes that crash during building or fail to meet inference accuracy standards are excluded, and the eligible engine with the fastest measured end-to-end performance is selected. The vendor reported performance improvements ranging from 1.14x to 1.63x (normalized against the default route) across ten evaluation workloads, but these are self-test results from NVIDIA without independent reproduction.
Performance-accuracy entanglement is a practical difficulty in inference optimization. The authors use a thought experiment to illustrate how the Global Performance Tuner disentangles them: imagine a simple neural network model with four locations for inserting quantization nodes (Q/DQ), each controlled by a binary knob. The compiler thus provides 2^4 = 16 candidate solutions. The solution with all quantizations inserted has the best performance but may not meet accuracy requirements; the solution with no insertions has the best accuracy but worst performance. It is difficult to predict in advance which intermediate solutions satisfy both requirements.
The solution is to introduce an accuracy tolerance threshold ε. When ε is zero, candidate outputs must be bit-wise identical to reference outputs, meaning only the solution with no low-precision substitutions is eligible. When ε is positive infinity, the accuracy constraint is ineffective, and all 16 solutions are eligible. The authors further assume that if the solution space contained an "empty" solution that performed no useful computation and had a runtime of zero, a performance-only optimizer might select it—demonstrating that performance alone is not a sufficient optimization objective. As ε gradually tightens from positive infinity to zero, the set of eligible candidates shrinks—for example, from 16 to 14, 13, 8, 7, and finally 1. However, these numbers are illustrative; actual transition points depend on the model, reference data, and selected accuracy metrics.
Thus, the design principle of the Global Performance Tuner is: accuracy determines which candidate solutions are acceptable, and performance ranks the acceptable solutions among themselves. Developers define accuracy boundaries suitable for their applications, and the tuner identifies the fastest measured build route within those boundaries. The authors explicitly state that this does not eliminate underlying trade-offs but separates the roles of performance and accuracy in the optimization process, making performance-accuracy optimization a measurable, reproducible, and application-defined constrained optimization process.
Hu and Chai also provide usage advice, one point of which relates directly to applicability boundaries: the optimal build route is strictly bound to the model, input shapes, target GPU, software stack, and TensorRT and tuner versions used during the search. A route that improves performance in one deployment may be ineffective or even cause regression in another. Re-tuning is required after changing the model, shapes, hardware, CUDA or driver environment, or relevant TensorRT versions. Knowledge gained from previous tuning experiments remains valuable—if certain knobs repeatedly improve a class of related workloads, that experience can guide future search space design and help prioritize promising candidates, but should serve as priors for focused exploration, with final routes still requiring validation and measurement in the actual deployment environment.
This blog post serves as the second part of the TensorRT 11.1 Global Performance Tuner series, focusing on algorithm and architecture design explanations and best practices; the first part introduced scheduling challenges, development motivations, and the mathematical theory formalizing the problem. Author identity and conflict of interest: Jian Hu is a Senior System Software Engineer at NVIDIA TensorRT, and Lin Chai is a Senior Product Manager at NVIDIA TensorRT; both hold positions within the development team for this feature.