LiveKit Engineer Jesse Hall: How to Build Financial Voice Agents That Stay Stable When Interrupted
Jesse Hall of LiveKit explains engineering methods for building human-like voice agents for financial services, focusing on why model leaderboards mislead developers, measuring latency by time-to-first-audio and time-to-first-sentence rather than first-byte, using async tool calls to hide backend delays, and benchmarking with simulated callers.
In a video, LiveKit engineer Jesse Hall shares insights on building voice agents for financial services that feel human rather than robotic. He specifically points out that traditional model leaderboards are often misleading because they evaluate individual components, whereas developers ultimately ship a complete conversation experience.
Hall suggests that developers should focus on "time to first audio" and "time to first sentence," rather than the commonly tracked "first byte" or "first token." He also explains how to keep conversations flowing while slow back-end work runs by using asynchronous tool calls. Additionally, he discusses audio-aware turn detection, a roughly 1.5-second latency budget (where 500-600ms feels truly human), and how to benchmark your own stack with simulated callers.