LectureRechercheRadarCadre d'investissement
Connexion / Inscription
← Retour à la lecture
ActualitésSujet · AI合成受访者Matériel1 min de lectureRédaction DisconfirmAI

LLMs Fail as Synthetic Survey Users

A new benchmark reveals that LLMs cannot outperform traditional baselines when simulating human survey responses, often distorting segment differences.

2026-10-06 12:00 GMT+8

Large language models fail to outperform traditional non-LLM baselines when simulating human survey responses.

The study by Zihan Chen et al. tested four models across U.S. social attitudes (GSS) and cross-cultural values (WVS). It found that models systematically overestimate the predictive power of demographics, inflating between-segment gaps by two to four times in targeting tasks.

This suggests teams using LLM-generated 'synthetic users' for market or policy decisions may target the wrong segments. The findings come from a preprint and have not yet been peer-reviewed or independently reproduced.

Sources

  • arxiv.org↗
Sujet

AI合成受访者

AI合成受访者能否替代真人调查?最新研究划定了明确边界:在多数通用大模型场景下,其误差高达14至19个百分点,仅具诊断价值;但在特定优化框架或专用平台中,准确率可显著提升。争议核心在于“像人”不等于“可替代”,关键在于区分预测试、方向性比较…

6 étapes2 piècesMis à jour 2026-10-08 09:16

Comment on en est arrivé là

  1. 2026-07-23韩国研究提交arXiv预印本,评估基于Gemini和EXAONE的合成面板,发现平均绝对误差达14-19个百分点,结论为合成面板不能替代调查,仅具诊断价值。
Connexion / Inscription
LectureRechercheRadarCadre d'investissement
arxiv.org ↗
  • 2026-07-28《When Synthetic Users Fail》论文提交,建立跨域基准,发现LLM无法超越非LLM基线,且会过度决定人口统计特征,导致细分目标错误。arxiv.org ↗
  • 2026-09-19问卷派发布指南,提出将“像人”与“可替代”分开验证,定义三层风险用途及六步对照校验协议,强调子群方向反转是常见失败模式。wj.pro ↗
  • Ce qui reste non résolu

    • 专用合成框架(如Artificial Societies)的低误差结果是否能在其他独立研究中复现?
    • 对于小语种或低代表性人群,合成数据的偏差结构是否与英语/高收入国家一致?
    • 行业是否会形成统一的合成数据披露标准,以区分“假设生成”与“决策支持”?

    Également dans ce fil

    • Korean study finds AI synthetic respondents cannot yet replace real surveys2026-10-02
    Lire le sujet complet →
    Continuer la recherche →