Idiap and Partners Release DialFork: First Benchmark for Procedural Hallucination in Conversational Agents

A joint team from Idiap Research Institute, Telefónica, and other institutions has released DialFork, the first benchmark designed specifically to measure 'procedural hallucination' in conversational agents. The paper was submitted to arXiv on October 15, 2026.

Traditional LLM hallucination evaluations focus on factual errors in generated content. However, in scenarios like customer support, scheduling, or clinical intake, agents are expected to follow strict procedural logic. Procedural hallucination occurs when an agent produces individually fluent and factually plausible utterances while deviating from the intended conversational process, such as skipping required steps or repeating completed actions.

DialFork comprises 36,278 dialogues across 24 tasks and 12 domains, with construction validated by 30 human annotators. The team introduced FlowKL and FlowScore, two metrics that compare dialogues based on how they unfold rather than their content. Experiments show that FlowScore significantly outperforms all baselines across seven encoders at the dialogue level, demonstrating that semantic similarity is not a substitute for procedural similarity.

The results are currently self-reported by the authors and have not yet been independently reproduced. The dataset and code are available under the MIT license on GitHub.