ReadingResearchRadarInvestment framework
Sign in / Sign up
← Back to reading
NewsMaterial2 min readDisconfirmAI Editorial

Audible Applied Scientist Arent van Korlaar: Why General-Purpose Speech Synthesis Cannot Sustain an Audiobook

The Audible team identifies three specific failure points of general-purpose TTS designed for short interactions in the audiobook context: inability to distinguish characters, lack of narrative context, and voice drift across chapters. Their solution delegates character and dialogue attribution to an independent BookLLM component, uses natural language descriptions as stage directions to drive controllable TTS rendering, and locks character voices at the book level.

2026-10-23 23:20 GMT+8

Voice assistants are already sufficiently natural in seconds-long interactions, but audiobooks require the same voice to persist for ten to fifteen hours.

In a technical blog post published on October 23, 2026, Audible applied scientist Arent van Korlaar attributes the failure of general-purpose TTS in audiobooks to three specific issues: First, models cannot switch speaking styles between different characters like human narrators, making it difficult for listeners to distinguish narration from dialogue or identify which character speaks which lines; Second, general-purpose TTS processes individual sentences with minimal context, whereas audiobooks require understanding surrounding context to maintain narrative coherence; Third, general-purpose models struggle to maintain consistent character voices across the span of an entire book, often voicing a character markedly differently in chapter one than in chapter ten.

Their solution is not to train a larger general-purpose TTS model, but to separate the architecture. The system consists of two components: BookLLM reads the entire book, builds a cast list, determines the attribution of each line of dialogue, and writes natural language annotations describing how each sentence should be performed; the controllable TTS model then synthesizes speech sentence by sentence based on these inputs. Because character voices are locked at the book level, the same character remains consistent from chapter one to chapter ten.

Sign in / Sign up
ReadingResearchRadarInvestment framework

BookLLM was tested on 25 English and Spanish mystery, sci-fi, and romance novels. Ground truth was established by two human annotators, achieving 97% accuracy in English and 94% in Spanish for character and dialogue attribution. In MUSHRA tests (a standardized listener-rating method scoring 0–100), character voice differentiation improved by 3.7 points, with eight out of ten character voices across both English and Spanish showing statistically significant improvements in listening experience. These are self-reported results by the Audible team; the number of test listeners and blind test details were not fully disclosed in the blog post.

The core logic of this architectural choice is that the problem with audiobooks is not insufficient speech naturalness, but rather giving different characters distinct, mutually discernible, and globally consistent voices throughout the book, while layering immediate emotional expression onto each line of dialogue.

Regarding data synthesis, the team faced a lack of expressiveness in training data—existing TTS training corpora mostly feature neutral tones. They expanded through two paths: first, using expert models such as pitch extraction, speaking rate extraction, and emotion detection to extract numerical features from audio, which are then fed into an LLM to generate speech descriptions; second, training a speech description model that takes speech as input and outputs prosodic feature descriptions, iteratively generating synthetic data in alternation with the controllable TTS model.

Sources

  • amazon.scienceAmazon Science↗
Continue researching →