Lambda et al. Release TurboPairFormer: 1.73x Faster OpenFold3 Training with Reproducible Gradients
Lambda, Stevens Institute of Technology, and the OpenFold team released TurboPairFormer, an open-source GPU kernel library that speeds up OpenFold3 protein folding model training by up to 1.73x while ensuring bitwise reproducible gradients.
Lambda, Stevens Institute of Technology, and the OpenFold team released TurboPairFormer on November 13, 2026, an open-source GPU kernel library designed to optimize the training efficiency of the biomolecular structure prediction model OpenFold3. The tool restructures the computationally expensive "triangle attention" operation, achieving up to 1.73x faster training on NVIDIA H100 and H200 GPUs.
According to the official blog post, TurboPairFormer improves performance through three key techniques: first, it recomputes attention probabilities only once during the backward pass, reducing redundant overhead; second, it employs output-residual compensation to capture rounding errors when converting from FP32 to BF16, lowering mean gradient error by 28% to 47%; third, it uses a fixed accumulation order to ensure gradients are bitwise identical across repeated runs, addressing non-determinism caused by floating-point atomic operations.
In tests on 16 H100 GPUs, TurboPairFormer demonstrated the lowest GPU computation time per optimizer step at all tested crop sizes. For instance, at a 768-token crop size, it was 1.73x faster than the original OpenFold3 Triton backend and 1.13x faster than NVIDIA cuEquivariance. Additionally, all gradients were fully reproducible across 600 fixed-input test cases, contributing to more stable large-scale training processes.