NVIDIA Employee Rick Grandy: Voice Agent Caching Strategy Reduces Response Latency to 0.72 Seconds

NVIDIA employee Rick Grandy documented the construction of MonsterHouse on the Hugging Face blog: a grumpy old pirate ship disguised as a garage door that converses with Halloween visitors via a real-time voice agent. His core engineering challenge was not model selection, but latency—a haunted house loses its authoritative presence if it complains too slowly.

The pipeline chains four models: Nemotron Speech ASR (nemotron-speech-streaming-en-0.6b, quantized to q8_0 GGUF format) for listening; Nemotron 3.5 Bolt E4B 4GB for reasoning; Magpie TTS Zeroshot for speaking; and Audio2Face-3D for lip-syncing. Bolt E4B is a 4-bit quantized open-source small language model; the author notes its footprint is smaller than Qwen3 30B-A3B and Qwen 3.6 35B-A3B, and it was selected after testing over ten models as the conversation core. Personality is provided by a LoRA adapter trained on over 5,000 multi-turn synthetic conversations, generated and evaluated by a local agent assisted by a large language model.

Latency management is the most transferable aspect of this work. The author reports two median data points: the full pipeline (Bolt Q4 plus TTS synthesis) has a median end-of-speech-to-audio output latency of 1.70 seconds, which drops to 0.72 seconds when using pre-cached responses. Two mechanisms jointly compress perceived wait time: first, the pipeline releases speakable text segments before the complete answer is generated, allowing the character to begin speaking while the rest of the sentence is still being produced; second, common greetings and candy jokes are pre-recorded as audio, bypassing both language model inference and speech synthesis steps. The author also notes that voice endpoint detection currently waits for 300 milliseconds of silence, which itself constitutes part of the latency floor.