독서리서치레이더투자 프레임워크
로그인 / 회원가입
← 읽기로 돌아가기
뉴스주제 · AWS语音合成优化중대3 분 소요DisconfirmAI 편집부

AWS Blog: Replacing TTS Auto-Scaling Guesses with Two Measured Values

For the non-autoregressive text-to-speech model deployed on Amazon Web Services' SageMaker AI platform in the United States, the load bottleneck is concurrent requests rather than throughput. The blog proposes a four-number model: select a latency SLO and overflow ratio, then measure the single-instance concurrency ceiling Q_max and end-to-end scaling time T_total; all other scaling parameters are derived arithmetically. This method is not limited to TTS and applies to latency-sensitive, concurrency-limited real-time inference endpoints on SageMaker AI.

2026-10-23 00:30 GMT+8

When traffic peaks arrive, text-to-speech (TTS) requests begin queuing, and latency breaches the service level objective (SLO)—even though auto-scaling policies were supposed to prevent this. The problem lies in the fact that the thresholds these policies rely on are often guessed based on experience. If guessed too conservatively, GPU instances sit idle, costing money on every invoice without triggering any alerts. If guessed too aggressively, each instance carries more concurrent requests than its latency promise allows. Everything looks normal in the console until a real traffic spike pushes queueing time past the SLO, exposing the failure at the moment most callers are listening.

Why can't TTS use the same scaling metrics as large language models? Most public guidelines target short synchronous request/response calls or LLM inference, where the natural metric is tokens generated per second. However, non-autoregressive models like Kokoro-82M produce full audio from a single forward pass, lacking the token-by-token generation loop needed to attach such metrics. What determines whether a TTS instance is overloaded is simpler: how many requests are currently queued and executing. Therefore, this solution uses SageMaker AI's native ConcurrentRequestsPerModel metric instead of custom throughput measures.

The core method is called the four-number model: operators select two parameters—the latency SLO (3,000 ms p95 time-to-first-audio-byte in this case) and an overflow ratio (1.25x, allowing traffic to grow by 25% within one scaling cycle without breaching the SLO)—then perform two measurements on the live endpoint. First is Q_max: using a closed-loop load ladder, increase concurrency step-by-step until p95 latency breaches the SLO, taking the last concurrency value that met the SLO. For Kokoro-82M on ml.g5.xlarge, Q_max was measured at 50. Second is T_total: force a scale-out event, timing each stage from capacity request to healthy traffic recovery. In this case, pure provisioning took 260.2 seconds; adding the upper bound of policy detection lag (60 seconds), the planned T_total is 320.2 seconds.

로그인 / 회원가입
독서리서치레이더투자 프레임워크

With Q_max and T_total, scaling thresholds and cooldowns are derived arithmetically: Scale-out threshold C_scale_max = (1−h) × Q_max = 0.75 × 50 = 37.5; Scale-in threshold C_scale_min = (1−2h) × Q_max = 0.50 × 50 = 25. The scale-out cooldown is short (30 seconds) because the cost of adding one instance is low. The scale-in cooldown is long (approximately 3 times T_total, i.e., 960 seconds) because if traffic rises again after removing an instance, re-scaling requires the full 320 seconds. The container’s own admission control hard-limits queue depth at Q_max=50, returning a 503 rejection for excess requests rather than letting them queue to inevitable timeout.

The key insight of this method extends beyond TTS. The blog notes that for latency-sensitive, concurrency-bound real-time inference endpoints on SageMaker AI—such as an image embedding endpoint keeping connections open during GPU forward passes—the bottleneck is also concurrent in-flight requests, not throughput metrics borrowed from other workload shapes. The derivation logic and five-step toolchain (drift, qmax, ttotal, plan, validate) can be migrated directly. However, specific values differ by model and instance type: Q_max=50 and T_total=320 seconds apply only to Kokoro-82M on ml.g5.xlarge; different models or machine types require new measurements.

출처

  • aws.amazon.comAmazon Web Services↗
주제

AWS语音合成优化

Amazon Web Services (AWS) 近期发布了两篇技术博客,详细阐述了如何在 Amazon SageMaker AI 平台上优化文本转语音(TTS)模型的部署性能与延迟。核心突破在于摒弃传统的猜测性自动扩缩容参数,转而采用基…

2 단계2 건업데이트됨 2026-10-23 05:04

현재까지의 과정

  1. 2026-10-23AWS 发布博客《Stop guessing auto scaling targets for TTS on Amazon SageMaker AI》,提出用两个实测值(Q_max 和 T_total)替代自动扩缩容的猜测参数,适用于延迟敏感、并发受限的实时推理端点。aws.amazon.com ↗
  2. 2026-10-23AWS 发布博客《Stream low-latency speech from AI agents using Kokoro TTS on Amazon SageMaker AI》,展示如何通过将逐句流式输入接入双向 TTS 端点,在不增加并行度的前提下将语音智能体首音频延迟降低 2.4 至 3.5 倍。aws.amazon.com ↗

아직 해결되지 않은 문제

  • 该四数模型(选定延迟SLO、溢出比率、Q_max、T_total)在其他非Kokoro系列的自回归TTS模型上是否同样适用?
  • 除了Kokoro-82M,AWS是否计划为其他主流开源TTS模型提供类似的官方扩缩容基准测试工具或指南?

이 스레드의 다른 내용

  • AWS Engineering Team: Voice Agents Don't Need Concurrent Synthesis, Just Immediate Startup on the First Sentence2026-10-23
전체 주제 읽기 →
리서치 계속하기 →