ReadingResearchRadarInvestment framework
Sign in / Sign up
← Back to reading
NewsMaterial3 min readDisconfirmAI Editorial

Speech-to-Text Company AssemblyAI: Distributing Across GPU Instance Families with Queue Retries Reduces Spot Inference Costs by 70%

AssemblyAI runs thousands of GPU instances on AWS for speech-to-text inference. Its key finding is that distributing demand across three GPU instance families—g4dn, g5, and g6—is the most effective lever for reducing Spot interruption rates. Combined with an SQS queue retry mechanism and a custom AMI that completes creation in approximately two minutes, Spot interruptions are transformed from failures into routine events, achieving a 70% reduction in inference costs.

2026-11-14 03:35 GMT+8

Speech-to-text inference is a GPU-intensive workload. Each audio file triggers a deep learning pipeline comprising acoustic models, language models, and post-processing stages, running on NVIDIA GPUs to meet latency requirements. At scale, this means thousands of GPU instances are running continuously. AssemblyAI needed a way to run these workloads at scale while maintaining availability and latency commitments; using On-Demand Instances exclusively was not cost-effective.

Spot Instances offer significant discounts, but Amazon Elastic Compute Cloud (Amazon EC2) can reclaim instances with a two-minute notice when capacity is needed elsewhere. The challenge was: how to absorb Spot interruptions gracefully while maintaining service level agreement (SLA) commitments? AssemblyAI’s answer involves four pillars working in concert: ECS capacity providers mixing Spot and On-Demand instances, diversification of Spot across multiple GPU instance families, a queue-based retry architecture, and multi-region deployment to satisfy data residency requirements.

The most transferable insight is instance family diversification. AssemblyAI’s infrastructure spans three EC2 GPU instance families: g4dn, equipped with NVIDIA T4 GPUs, provides a cost-effective baseline for standard speech recognition models; g5, equipped with NVIDIA A10G GPUs, offers higher throughput for more demanding model architectures; and g6, featuring NVIDIA L4 GPUs, delivers improved performance-per-dollar for larger models, including large language model inference. By spreading demand across instance families and sizes, AssemblyAI draws from a significantly larger pool of available Spot capacity. These three families tap into independent capacity pools, so when one family experiences capacity pressure, the others remain available. The article explicitly states that operating at this scale, diversifying across instance families and sizes is the most effective lever for minimizing Spot interruption rates.

Sign in / Sign up
ReadingResearchRadarInvestment framework

The recovery path during interruptions is equally critical. When a Spot instance receives a two-minute interruption notice, tasks that can complete within the window finish normally; tasks requiring more time are returned to the SQS queue, becoming immediately available to other workers. Another healthy GPU instance may pick up the message and complete the inference on a different instance type or in a different Availability Zone. To shorten recovery time, AssemblyAI creates custom AMIs containing model artifacts and container images, which take approximately two minutes to build, allowing new instances to start inference directly without pulling images or downloading models. For end customers, the impact of a Spot interruption is a slight increase in processing time, but the transcription ultimately completes successfully and is delivered as expected.

At the capacity scheduling level, AssemblyAI configures multiple capacity providers within a single capacity provider strategy in each ECS cluster. An On-Demand capacity provider serves as an always-on baseline, while a Spot capacity provider spans different GPU instance families and Availability Zones to handle demand spikes. Each capacity provider is associated with an EC2 Auto Scaling group, and ECS managed scaling automatically adjusts the number of GPU instances based on capacity provider weights and pending inference tasks. When the task queue grows, the Spot capacity provider requests additional instances from the diversified pool; when demand drops, instances scale in gracefully.

Boundaries to emphasize: all performance data—including the 70% cost savings, approximately fourfold year-over-year growth in compute demand, and the approximately two-minute AMI creation time—are self-reported by AssemblyAI and AWS. The 70% figure compares against On-Demand pricing; the article notes that running entirely on On-Demand Instances would be cost-prohibitive but does not disclose the mixed configuration AssemblyAI previously operated. This pattern applies to short-lived, independently retriable GPU computing tasks; training jobs requiring long-term continuous state maintenance or inference scenarios maintaining context across requests do not fit the queue-retry assumption. Furthermore, Spot availability for g4dn, g5, and g6 in specific European regions is subject to AWS capacity fluctuations, meaning the effectiveness of the diversification strategy depends on the actual independence of these capacity pools.

Sources

  • aws.amazon.comAmazon Web Services↗
Continue researching →