AWS Blog: Replacing TTS Auto-Scaling Guesses with Two Measured Values
For the non-autoregressive text-to-speech model deployed on Amazon Web Services' SageMaker AI platform in the United States, the load bottleneck is concurrent requests rather than throughput. The blog proposes a four-number model: select a latency SLO and overflow ratio, then measure the single-instance concurrency ceiling Q_max and end-to-end scaling time T_total; all other scaling parameters are derived arithmetically. This method is not limited to TTS and applies to latency-sensitive, concurrency-limited real-time inference endpoints on SageMaker AI.
When traffic peaks arrive, text-to-speech (TTS) requests begin queuing, and latency breaches the service level objective (SLO)—even though auto-scaling policies were supposed to prevent this. The problem lies in the fact that the thresholds these policies rely on are often guessed based on experience. If guessed too conservatively, GPU instances sit idle, costing money on every invoice without triggering any alerts. If guessed too aggressively, each instance carries more concurrent requests than its latency promise allows. Everything looks normal in the console until a real traffic spike pushes queueing time past the SLO, exposing the failure at the moment most callers are listening.
Why can't TTS use the same scaling metrics as large language models? Most public guidelines target short synchronous request/response calls or LLM inference, where the natural metric is tokens generated per second. However, non-autoregressive models like Kokoro-82M produce full audio from a single forward pass, lacking the token-by-token generation loop needed to attach such metrics. What determines whether a TTS instance is overloaded is simpler: how many requests are currently queued and executing. Therefore, this solution uses SageMaker AI's native ConcurrentRequestsPerModel metric instead of custom throughput measures.
The core method is called the four-number model: operators select two parameters—the latency SLO (3,000 ms p95 time-to-first-audio-byte in this case) and an overflow ratio (1.25x, allowing traffic to grow by 25% within one scaling cycle without breaching the SLO)—then perform two measurements on the live endpoint. First is Q_max: using a closed-loop load ladder, increase concurrency step-by-step until p95 latency breaches the SLO, taking the last concurrency value that met the SLO. For Kokoro-82M on ml.g5.xlarge, Q_max was measured at 50. Second is T_total: force a scale-out event, timing each stage from capacity request to healthy traffic recovery. In this case, pure provisioning took 260.2 seconds; adding the upper bound of policy detection lag (60 seconds), the planned T_total is 320.2 seconds.