Voice AI company Deepgram says Nova-3 ported to AWS Trainium2 processes 11–70% more audio per dollar on batch inference

Voice AI company Deepgram processes quadrillions of audio tokens a year for more than 200,000 developers and 1,400 organizations. Its offline batch workloads—contact-center analytics, medical transcription and similar scenarios—are extremely sensitive to inference cost. In October 2026, Deepgram and the AWS Forward Deployed Engineering team published a complete engineering record of porting the Nova-3 speech-to-text model to Trainium2 chips, self-reporting that a single EC2 trn2.3xlarge instance processed 11–70% more audio per dollar.

The port was not a simple compilation swap. The team first ran Nova-3 in eager mode on Trainium2 via TorchNeuron and checked which operations fell back to the CPU, then compiled with torch.compile, producing separate static-shape graphs for the encoder and decoder. The compiled path ran every model operation on Trainium2 with no CPU fallbacks detected. Numerical accuracy was validated at two levels—layer-level output consistency against CPU and end-to-end Word Error Rate—ensuring alignment with Deepgram's reference values.

The real gains came from kernel-level optimization. The team adopted a flywheel strategy: enable an NKILib pre-built kernel, profile with Neuron Explorer, then tune or hand-write a replacement based on profiling insights. Three key findings each addressed a specific bottleneck. The reference kernel for the feed-forward network layer reloaded the full set of projection weights from off-chip memory for every token tile, making memory bandwidth the limiting factor as token count grew; the hand-tuned kernel loads each weight tile once and reuses it across multiple token tiles, achieving 2.0× lower latency at batch size 96. The experimental NKILib kernel for depthwise 1D convolution framed the operation as an implicit GEMM, leaving Tensor Engine utilization low; the team rewrote it as a broadcast multiply-accumulate loop on the Vector Engine, reaching full engine utilization for a 1.5× speedup at batch size 96. The conv2d layers hit a compilation bottleneck beyond batch size 32, preventing single-graph compilation; since NKILib had no conv2d kernel, the team wrote one from scratch in NKI, unlocking batch sizes above 32 and achieving up to 50× speedup on the convolution module at batch sizes ≤32.