Korean study finds AI synthetic respondents cannot yet replace real surveys
Peer-reviewed testing put synthetic respondent error at 14-19 percentage points; the panels are diagnostic tools, not survey substitutes.
ImportanceLocalEvidenceE3 inspectableWrite-upQuick
A Korean research team has published a validation in IEEE Access: using LLM personas to simulate survey respondents produces errors too large to replace real questionnaires.
The study had Gemini 3.5 Flash and EXAONE each generate about 8,000 Korean personas, answer eight digital and AI service-use items, and compared the responses with weighted estimates from the Korea Media Panel Survey. Overall mean absolute error was 14-19 percentage points, and segment error across five axes was 14-18 percentage points, with the two models showing different bias patterns.
Even after calibrating on 30% of the real data, which cut error to about 5 percentage points, direct estimation from that same real subsample was more accurate (3.6 percentage points). The authors conclude that synthetic panels are diagnostic tools, not survey substitutes.