Voice AI company Deepgram says Nova-3 ported to AWS Trainium2 processes 11–70% more audio per dollar on batch inference
Deepgram worked with the AWS Forward Deployed Engineering team to port its Nova-3 speech-to-text model to AWS Trainium2 chips for offline batch inference. The post describes the full optimization flow from PyTorch compilation validation, NKI kernel replacement and hand-tuning, to a data-parallel Rust runtime, and self-reports processing 11–70% more audio per dollar on trn2.3xlarge instances.
ImportanceLocalEvidenceE2 unreplicatedWrite-upQuick
Voice AI company Deepgram processes quadrillions of audio tokens a year for more than 200,000 developers and 1,400 organizations. Its offline batch workloads—contact-center analytics, medical transcription and similar scenarios—are extremely sensitive to inference cost. In October 2026, Deepgram and the AWS Forward Deployed Engineering team published a complete engineering record of porting the Nova-3 speech-to-text model to Trainium2 chips, self-reporting that a single EC2 trn2.3xlarge instance processed 11–70% more audio per dollar.
The port was not a simple compilation swap. The team first ran Nova-3 in eager mode on Trainium2 via TorchNeuron and checked which operations fell back to the CPU, then compiled with torch.compile, producing separate static-shape graphs for the encoder and decoder. The compiled path ran every model operation on Trainium2 with no CPU fallbacks detected. Numerical accuracy was validated at two levels—layer-level output consistency against CPU and end-to-end Word Error Rate—ensuring alignment with Deepgram's reference values.
The real gains came from kernel-level optimization. The team adopted a flywheel strategy: enable an NKILib pre-built kernel, profile with Neuron Explorer, then tune or hand-write a replacement based on profiling insights. Three key findings each addressed a specific bottleneck. The reference kernel for the feed-forward network layer reloaded the full set of projection weights from off-chip memory for every token tile, making memory bandwidth the limiting factor as token count grew; the hand-tuned kernel loads each weight tile once and reuses it across multiple token tiles, achieving 2.0× lower latency at batch size 96. The experimental NKILib kernel for depthwise 1D convolution framed the operation as an implicit GEMM, leaving Tensor Engine utilization low; the team rewrote it as a broadcast multiply-accumulate loop on the Vector Engine, reaching full engine utilization for a 1.5× speedup at batch size 96. The conv2d layers hit a compilation bottleneck beyond batch size 32, preventing single-graph compilation; since NKILib had no conv2d kernel, the team wrote one from scratch in NKI, unlocking batch sizes above 32 and achieving up to 50× speedup on the convolution module at batch sizes ≤32.