독서리서치레이더투자 프레임워크
로그인 / 회원가입
← 읽기로 돌아가기
뉴스주제 · AssemblyAI중대3 분 소요DisconfirmAI 편집부

Speech-to-Text Company AssemblyAI: Distributing Across GPU Instance Families with Queue Retries Reduces Spot Inference Costs by 70%

AssemblyAI runs thousands of GPU instances on AWS for speech-to-text inference. Its key finding is that distributing demand across three GPU instance families—g4dn, g5, and g6—is the most effective lever for reducing Spot interruption rates. Combined with an SQS queue retry mechanism and a custom AMI that completes creation in approximately two minutes, Spot interruptions are transformed from failures into routine events, achieving a 70% reduction in inference costs.

2026-11-14 03:35 GMT+8

Speech-to-text inference is a GPU-intensive workload. Each audio file triggers a deep learning pipeline comprising acoustic models, language models, and post-processing stages, running on NVIDIA GPUs to meet latency requirements. At scale, this means thousands of GPU instances are running continuously. AssemblyAI needed a way to run these workloads at scale while maintaining availability and latency commitments; using On-Demand Instances exclusively was not cost-effective.

Spot Instances offer significant discounts, but Amazon Elastic Compute Cloud (Amazon EC2) can reclaim instances with a two-minute notice when capacity is needed elsewhere. The challenge was: how to absorb Spot interruptions gracefully while maintaining service level agreement (SLA) commitments? AssemblyAI’s answer involves four pillars working in concert: ECS capacity providers mixing Spot and On-Demand instances, diversification of Spot across multiple GPU instance families, a queue-based retry architecture, and multi-region deployment to satisfy data residency requirements.

The most transferable insight is instance family diversification. AssemblyAI’s infrastructure spans three EC2 GPU instance families: g4dn, equipped with NVIDIA T4 GPUs, provides a cost-effective baseline for standard speech recognition models; g5, equipped with NVIDIA A10G GPUs, offers higher throughput for more demanding model architectures; and g6, featuring NVIDIA L4 GPUs, delivers improved performance-per-dollar for larger models, including large language model inference. By spreading demand across instance families and sizes, AssemblyAI draws from a significantly larger pool of available Spot capacity. These three families tap into independent capacity pools, so when one family experiences capacity pressure, the others remain available. The article explicitly states that operating at this scale, diversifying across instance families and sizes is the most effective lever for minimizing Spot interruption rates.

로그인 / 회원가입
독서리서치레이더투자 프레임워크

The recovery path during interruptions is equally critical. When a Spot instance receives a two-minute interruption notice, tasks that can complete within the window finish normally; tasks requiring more time are returned to the SQS queue, becoming immediately available to other workers. Another healthy GPU instance may pick up the message and complete the inference on a different instance type or in a different Availability Zone. To shorten recovery time, AssemblyAI creates custom AMIs containing model artifacts and container images, which take approximately two minutes to build, allowing new instances to start inference directly without pulling images or downloading models. For end customers, the impact of a Spot interruption is a slight increase in processing time, but the transcription ultimately completes successfully and is delivered as expected.

At the capacity scheduling level, AssemblyAI configures multiple capacity providers within a single capacity provider strategy in each ECS cluster. An On-Demand capacity provider serves as an always-on baseline, while a Spot capacity provider spans different GPU instance families and Availability Zones to handle demand spikes. Each capacity provider is associated with an EC2 Auto Scaling group, and ECS managed scaling automatically adjusts the number of GPU instances based on capacity provider weights and pending inference tasks. When the task queue grows, the Spot capacity provider requests additional instances from the diversified pool; when demand drops, instances scale in gracefully.

Boundaries to emphasize: all performance data—including the 70% cost savings, approximately fourfold year-over-year growth in compute demand, and the approximately two-minute AMI creation time—are self-reported by AssemblyAI and AWS. The 70% figure compares against On-Demand pricing; the article notes that running entirely on On-Demand Instances would be cost-prohibitive but does not disclose the mixed configuration AssemblyAI previously operated. This pattern applies to short-lived, independently retriable GPU computing tasks; training jobs requiring long-term continuous state maintenance or inference scenarios maintaining context across requests do not fit the queue-retry assumption. Furthermore, Spot availability for g4dn, g5, and g6 in specific European regions is subject to AWS capacity fluctuations, meaning the effectiveness of the diversification strategy depends on the actual independence of these capacity pools.

출처

  • aws.amazon.comAmazon Web Services↗
주제

AssemblyAI

语音转写公司AssemblyAI公布其基于Amazon Web Services (AWS)的推理成本优化方案,通过混合使用Amazon EC2 Spot与按需实例,实现推理计算成本降低70%。该架构核心在于跨g4dn、g5、g6三个GPU…

2 단계1 건업데이트됨 2026-11-18 08:46

현재까지의 과정

  1. 2026-11-13AWS机器学习博客发布文章,披露AssemblyAI利用Amazon EC2 Spot实例优化语音转写推理成本的架构细节。aws.amazon.com ↗
  2. 2026-11-14Disconfirm收录该报道,确认AssemblyAI通过跨GPU实例族分散需求及队列重试机制,实现70%推理成本节省。aws.amazon.com ↗

아직 해결되지 않은 문제

  • 该架构在应对极端Spot容量短缺时的实际延迟波动范围是多少?
  • 除AWS外,其他云服务商是否提供类似的跨实例族自动调度能力以复现此成本效益?
전체 주제 읽기 →
리서치 계속하기 →