Preprint: Separate Answer Call Improves Evidence Following Over Reasoning Continuation

A preprint published on arXiv titled "When Reasoning Traces Conflict with Evidence: How Answer Configuration Shapes Evidence Following" investigates how large language models handle conflicts between their internal reasoning traces and external evidence. The study was conducted by Shaoang Li, Yanhang Shi, and Jian Li.

The researchers found that retaining a reasoning trace at the time of answering plays two distinct roles: it can improve accuracy, but it can also compete with current evidence when the two disagree. By testing three model families (gpt-oss, Gemma4, Qwen3) on HotpotQA and MuSiQue datasets, they discovered that using a separate answer call—rather than continuing the model's own thinking process—raises evidence-following by 29.9 to 60.6 percentage points.

Although removing the trace lowers accuracy in both configurations, the data shows that continuation often repeats stale conclusions in most conflict scenarios, whereas the separate call follows current evidence far more frequently. This suggests that benefiting from a trace and deferring to it are distinct properties.

The study recommends that because trace deletion alone cannot reveal which source an answer follows, evaluations should report both properties together with the answer configuration. This finding provides important reference for optimizing agent decision logic in Retrieval-Augmented Generation (RAG) systems.