Study Finds Rubric Content Drives LLM Judge Pass Rates More Than Model Choice
A new preprint shows that holding the judge model fixed, rubric content can swing correct output pass rates from 2% to 100%, while model swaps cause only minor shifts; self-review tends to delete effective checks.
ImportanceMaterialEvidenceE2 unreplicatedWrite-upQuick
The arXiv preprint "Rubrics Carry the Standard: Shapley Attribution for Checklist-Based LLM Judges" challenges two common assumptions in Large Language Model (LLM) evaluation: that leniency or strictness is primarily a property of the judge model, and that asking a model to review its own rubric improves it.
In controlled experiments using human-labeled datasets SummEdits and RAGTruth, researchers found that keeping the judge model and outputs constant, varying the rubric content caused the pass rate of correct outputs to fluctuate dramatically between 2% and 100%. In contrast, changing the judge model or the rendering of identical checks resulted in only a few percentage points of difference.
The study further demonstrates that model self-review drifts rubrics toward the reviser's own standards, adding unstated requirements and deleting checks that were validated by human labels. To address this, the authors propose a Shapley attribution method to quantify each check's contribution, showing that self-review systematically removes unique catches while retaining those causing false failures.