Using rubrics to score large language model outputs, or using scores generated by rubrics as reward signals for training models, is now a common practice. But a question rarely asked directly is: Are the rubrics themselves reliable?
In a blog post published in October 2026, Snorkel AI Senior Research Scientist Ankit Aich and colleagues introduced their proposed RIFT (RubrIc Failure mode Taxonomy) framework. This framework treats rubrics as measurement instruments, evaluating them along two dimensions from measurement theory: reliability (whether grading is reproducible) and content validity (whether grading supports the intended interpretation of the score).
Aich et al. note that past assessments of rubric quality typically relied on downstream evidence—such as the agreement rate between automated judges and human raters, or performance gains observed in models trained on rubric-derived rewards. Both approaches confound the rubric with the model that applies it, making it impossible to detect a rubric that is internally consistent and clearly expressed yet misaligned with the task objectives.
RIFT comprises two components: first, quality signals computed from the rubric text, where a simulated expert panel judges whether each criterion is essential to the task, compares criteria against task topics and objectives to assess coverage, and checks whether point allocations align with importance rankings; second, a taxonomy of nine failure modes, including subjective wording, ungrounded verification, non-atomicity, missing coverage, misalignment, rigidity, redundancy, hackability, and miscalibrated weighting.
To validate the signals, the authors conducted controlled injection experiments: injecting each failure mode at known severity levels into 60 expert-authored rubrics, ensuring the true defect was known for every variant. A multinomial logistic regression classifier identified the injected modes based on signal changes, achieving 75.0% accuracy under leave-one-task-out cross-validation (random baseline 12.5%); the same rubrics submitted to an LLM judge achieved 56.7% accuracy.
At the application level, the team applied RIFT to 147 tasks across five public benchmarks (27 tasks in finance, and 30 each in the other four domains), covering quantitative analysis of corporate financial reports, perioperative clinical management, debugging and implementation for coding agents, multi-turn consulting conversations, and open-ended reasoning dilemmas. The core finding is that mechanical quality is generally sound, while validity issues are prominent. Non-atomic standards appeared in 7% of tasks, and subjective wording in 10%—these were the least frequent. In contrast, missing coverage appeared in 50% of tasks, misalignment in 48%, ungrounded verification in 37%, and redundancy in 35%.
More notably, cross-domain patterns emerged: failure types correlated with the verifiability of answers. Finance and software engineering domains failed primarily on ungrounded verification (74% and 57%), because their standards require specific values or behaviors but often omit reference baselines; medicine, professional advice, and reasoning domains failed primarily on misalignment (73%, 57%, and 70%), because judgment-based tasks often embed assumptions leading to conclusions different from what the task requires.