ReadingResearchRadarInvestment framework
Sign in / Sign up
← Back to reading
NewsMaterial4 min readDisconfirmAI Editorial

AWS Engineering Team: Voice Agents Don't Need Concurrent Synthesis, Just Immediate Startup on the First Sentence

The AWS SageMaker AI team's practical tests show that feeding sentence-by-sentence streaming input into a bidirectional TTS endpoint can reduce the first audio latency of voice agents by 2.4 to 3.5 times without increasing concurrency. The article also quantifies the 33% payload inflation caused by base64 encoding in SSE and MP3 frame alignment issues.

2026-10-23 01:45 GMT+8

The most common latency experience issue for voice agents is not that model generation is too slow, but that synthesis starts too late. A user asks a question, the language model is already generating a response token by token, but the text-to-speech module waits until the entire response is complete before it begins working. A single-sentence response might be generated in 600 milliseconds, but a full multi-paragraph response takes five seconds, and the integration logic waits for those five seconds. During these five seconds, the user hears nothing.

A technical solution released by the AWS SageMaker AI team on October 22, 2026, points out that the key to solving this problem is not running synthesis and generation in parallel, but changing the timing of synthesis startup: as soon as the language model produces a complete sentence, send that sentence immediately to the TTS model, without waiting for subsequent content. They deployed Kokoro-82M (with only 82 million parameters) on a SageMaker AI real-time endpoint, maintaining a persistent connection via a bidirectional WebSocket stream, sending text sentence by sentence and receiving audio sentence by sentence.

There is a potential misunderstanding here. The team explicitly clarifies that this is not concurrent synthesis—the loop of the bidirectional processor is strictly sequential: receive a message, synthesize its audio, send the audio, send a completion signal, then receive the next message. The audio for the second sentence does not begin generating while the first sentence is still being transmitted. What incremental sending actually achieves is that synthesis for the first sentence begins the instant it exists, rather than waiting for the full response. For multi-sentence responses, there is a genuine generation time gap between the first and last sentences; this gap represents the entire latency benefit.

Sign in / Sign up
ReadingResearchRadarInvestment framework

Isolated measurements show that sending only the first sentence (rather than the full four-sentence text) reduced the time to first audio byte at the TTS transport layer from 329.3 milliseconds to 156.7 milliseconds, a relative improvement of 52.4%. In end-to-end comparisons, the bidirectional streaming mode was 2.4 times faster than batch mode (local container) and 3.5 times faster (deployed AgentCore runtime). The gap actually widens when real network hops are added, because the streaming mode absorbs network overhead within the overlap window of generation and synthesis, whereas every serial wait step in batch mode directly exposes network latency.

There is a second cost at the transport layer. If the frontend is already streaming text via Server-Sent Events (SSE), the team measured the cost of stuffing base64-encoded audio into SSE events: payload inflation is approximately 33%, and the SSE delimiter (empty line) does not align with internal MP3 frame boundaries—in practice, four logical MP3 frames arrived as six transmission blocks, two of which were truncated mid-frame. They compared MP3 at 48kbps and AAC 32kbps fMP4 encoding, confirming with Amazon Transcribe that decoding quality showed no difference.

For projects with an existing SSE pipeline, the base64 approach is a pragmatic choice, at the cost of 33% extra payload and a buffer reassembly step. For scenarios building a voice path from scratch or prioritizing latency, the team recommends directly reusing binary audio and JSON control frames on the same bidirectional session—the distinguishing rule is: if a data block cannot be parsed as a valid JSON object, it is audio. This rule works because although raw PCM audio may start with the byte 0x7b (left curly brace), the entire PCM frame will almost never coincidentally constitute a valid JSON object.

All data comes from self-tests conducted by the AWS team on their own SageMaker AI endpoints and Bedrock AgentCore runtime; independent third-party reproduction was not provided. The 82-million-parameter scale of Kokoro-82M is suitable for running on a single mid-tier GPU instance, but the article does not provide subjective quality assessments of synthesized speech or quality comparisons with other TTS models. The example container pins an older version of PyTorch with a known security advisory (PYSEC-2026-2286); the team notes that an upgrade is required before formal use.

Sources

  • aws.amazon.comAmazon Web Services↗
Continue researching →