Study Finds LLM Financial Advice Swings Sharply with User Risk Answers; One Clarifying Question Mitigates Variance
A preprint study reveals that major LLMs shift stock allocation advice by an average of 10-16 percentage points when users change their risk tolerance answers; adding a single clarifying question reduces this swing from 33.5 to 1.9 points.
ImportanceMaterialEvidenceE2 unreplicatedWrite-upQuick
The arXiv preprint "Same Person, Different Advice: Auditing LLM Financial Advice with Repeated Risk-Tolerance Responses" highlights that many AI financial advisers currently set stock allocations based largely on a single stated-risk question, despite the fact that human responses to such questions are often inconsistent. The research team tested seven large language models from five providers, generating one-turn allocation advice for 800 real household profiles. They found that when only the user's risk answer changed, the recommended stock share moved by an average of 10 to 16 percentage points. If the answer changed substantially, the swing reached 24 to 35 percentage points. In contrast, repeating the same prompt twice resulted in a movement of only 0.6 to 4.2 percentage points.
The study notes that this volatility driven by input noise is most pronounced among clients under 40. The authors propose that if this response pattern represents how models map attitude to advice, discounting unreliable answers according to classical test theory could theoretically reduce swings by a factor of 2.4 to 3.3. Furthermore, experiments showed that while noting the answer was "noisy" steadied some models, it caused them to over-rely on age factors and push extremely cautious clients toward generic middle allocations (e.g., advising 22% stocks instead of 4% for clients unwilling to take any risk).
The most actionable finding is that in a scripted test, adding just one clarifying question removed nearly all of the swing caused by a client subsequently correcting their answer—reducing the average variance from 33.5 percentage points to 1.9—while preserving the model's ability to respond to confirmed changes in preference. The researchers argue that same-person retest audits should become a standard check for AI advisers, and that handling unreliable answers is a matter of client autonomy as well as statistics.