독서리서치레이더투자 프레임워크
로그인 / 회원가입
← 읽기로 돌아가기
뉴스주제 · XYBench중대2 분 소요DisconfirmAI 편집부

Ai2 Releases XYBench to Test If Large Models Can Identify User Misconceptions and Redirect to Real Goals

Ai2 and partners released XYBench, a benchmark evaluating AI assistants' handling of 'XY problems.' Results show mainstream models often focus on flawed premises rather than solving underlying needs, revealing a significant gap in intent understanding.

2026-11-14 01:02 GMT+8

Non-profit research institute Allen Institute for AI (Ai2), in collaboration with researchers from Carnegie Mellon University, NVIDIA, and Johns Hopkins University, has released XYBench, a new benchmark comprising over 8,000 questions designed to test how AI assistants respond to requests based on misconceptions. The benchmark targets the 'XY problem': a user wants to achieve goal X but assumes approach Y will work, thus asking for help with Y while ignoring the actual issue. For example, a user might ask how to keep frosting from sliding off cupcakes, when the real problem is that the cupcakes are too warm to frost.

The research team evaluated models including Claude Sonnet 4.6, GPT-5, Gemini 3.1 Pro, Qwen3.5-9B, and DeepSeek-V4 Pro. The results showed that while these models often suggested solutions, they tended to keep their answers focused on the user's original, mistaken request rather than steering them toward their underlying goal. Compared to human reference answers, all five models were less likely to steer users away from the mistaken approach and toward a solution suited to their goal.

XYBench measures model responses across three dimensions: pragmatic solution (whether it addresses both the stated request and the intended goal), pragmatic emphasis (whether the answer focuses on the stated request or the intended goal), and misconception identification (whether it discourages the mistaken approach and guides toward the correct one). Experiments indicated that even when the user's potential goal was explicitly provided, models still underperformed humans in misconception identification, and sometimes performed worse, suggesting that detecting intent is only part of the challenge; models must also act on it when it diverges from the literal request.

로그인 / 회원가입
독서리서치레이더투자 프레임워크

The benchmark won the Best Resource Paper Award at EMNLP 2026. Data, code, and the paper are publicly available for researchers to evaluate whether prompting or training improvements can help models better identify misconceptions and offer more effective alternatives.

출처

  • huggingface.co↗
주제

XYBench

美国艾伦人工智能研究所(Ai2)联合卡内基梅隆大学等机构发布XYBench基准,旨在测试大语言模型能否识别用户提问中的“XY问题”并引导至真实目标。该基准包含8115个问题,涵盖编程与日常任务。实验显示,Claude Sonnet 4.6、…

3 단계1 건업데이트됨 2026-11-18 02:19

현재까지의 과정

  1. 2026-09-06Ai2团队提交关于XYBench的研究论文至arXiv,提出基于合作响应理论的评估范式。arxiv.org ↗
  2. 2026-11-13Ai2正式发布XYBench基准,公布由Stack Overflow、WikiHow等来源构成的8115个测试问题及评估代码。huggingface.co ↗
  3. 2026-11-13Ai2发布实验结果,指出被测模型在单轮对话中更倾向于回答字面请求(Y),而非用户实际目标(X)。allenai.org ↗

아직 해결되지 않은 문제

  • 模型在多轮对话或允许追问的场景下,识别和纠正用户误解的能力是否会显著提升?
  • 除了英语和程序性任务,XYBench在非英语文化背景及非结构化问题中的适用性如何?
  • 如何通过微调或提示工程优化模型,使其在生成回答时能主动平衡对字面请求的响应与对潜在目标的引导?
전체 주제 읽기 →
리서치 계속하기 →