AWS Blog: Replacing TTS Auto-Scaling Guesses with Two Measured Values
For the non-autoregressive text-to-speech model deployed on Amazon Web Services' SageMaker AI platform in the United States, the load bottleneck is concurrent requests rather than throughput. The blog proposes a four-number model: select a latency SLO and overflow ratio, then measure the single-instance concurrency ceiling Q_max and end-to-end scaling time T_total; all other scaling parameters are derived arithmetically. This method is not limited to TTS and applies to latency-sensitive, concurrency-limited real-time inference endpoints on SageMaker AI.
When traffic peaks arrive, text-to-speech (TTS) requests begin queuing, and latency breaches the service level objective (SLO)—even though auto-scaling policies were supposed to prevent this. The problem lies in the fact that the thresholds these policies rely on are often guessed based on experience. If guessed too conservatively, GPU instances sit idle, costing money on every invoice without triggering any alerts. If guessed too aggressively, each instance carries more concurrent requests than its latency promise allows. Everything looks normal in the console until a real traffic spike pushes queueing time past the SLO, exposing the failure at the moment most callers are listening.
Why can't TTS use the same scaling metrics as large language models? Most public guidelines target short synchronous request/response calls or LLM inference, where the natural metric is tokens generated per second. However, non-autoregressive models like Kokoro-82M produce full audio from a single forward pass, lacking the token-by-token generation loop needed to attach such metrics. What determines whether a TTS instance is overloaded is simpler: how many requests are currently queued and executing. Therefore, this solution uses SageMaker AI's native ConcurrentRequestsPerModel metric instead of custom throughput measures.