LectureRechercheRadarCadre d'investissement
Connexion / Inscription
← Retour à la lecture
ActualitésSujet · LLM裁判评分标准敏感性Matériel1 min de lectureRédaction DisconfirmAI

Study Finds Rubric Content Drives LLM Judge Pass Rates More Than Model Choice

A new preprint shows that holding the judge model fixed, rubric content can swing correct output pass rates from 2% to 100%, while model swaps cause only minor shifts; self-review tends to delete effective checks.

2026-10-20 12:00 GMT+8

The arXiv preprint "Rubrics Carry the Standard: Shapley Attribution for Checklist-Based LLM Judges" challenges two common assumptions in Large Language Model (LLM) evaluation: that leniency or strictness is primarily a property of the judge model, and that asking a model to review its own rubric improves it.

In controlled experiments using human-labeled datasets SummEdits and RAGTruth, researchers found that keeping the judge model and outputs constant, varying the rubric content caused the pass rate of correct outputs to fluctuate dramatically between 2% and 100%. In contrast, changing the judge model or the rendering of identical checks resulted in only a few percentage points of difference.

The study further demonstrates that model self-review drifts rubrics toward the reviser's own standards, adding unstated requirements and deleting checks that were validated by human labels. To address this, the authors propose a Shapley attribution method to quantify each check's contribution, showing that self-review systematically removes unique catches while retaining those causing false failures.

Sources

  • arxiv.org↗
Sujet
Connexion / Inscription
LectureRechercheRadarCadre d'investissement

LLM裁判评分标准敏感性

在利用大语言模型(LLM)作为自动裁判进行开放式输出评估时,业界长期存在一种假设:评估的宽松或严格程度主要取决于裁判模型本身,且可以通过让模型自我审查来优化评分标准。然而,最新研究【背景】表明,这一认知存在严重偏差。固定裁判模型和输出内容时…

3 étapes1 piècesMis à jour 2026-10-21 00:50

Comment on en est arrivé là

  1. 2026-10-18预印本《Rubrics Carry the Standard》发布,指出在固定裁判模型时,评分标准内容对通过率的影响远大于模型更换,且模型自审会导致标准漂移。arxiv.org ↗
  2. 2026-01-01论文《From Rubrics to Reliable Scores》提出Rulers框架,通过编译不可变的评分规范和证据锚定协议,解决LLM裁判的解释漂移和尺度错位问题。arxiv.org ↗
  3. 2025-06-01蚂蚁集团团队发表研究,定义“评分偏差”,发现即使是最先进的LLM也受评分提示中的顺序、ID和参考答案选择等细微扰动影响。link.springer.com ↗

Ce qui reste non résolu

  • 如何自动化地生成和维护不会导致漂移的“不可变”评分规范?
  • 在不同领域(如医疗、法律)的基准测试中,评分标准敏感性的阈值是否一致?
  • 现有的开源工具(如Rulers代码库)能否被工业界大规模部署以替代黑盒API裁判?
Lire le sujet complet →
Continuer la recherche →