Preprint Finds Input Context Matters More Than Judge Model Choice in LLM Evaluation

A preprint submitted on October 11, 2026, finds that in text evaluation using Large Language Models (LLMs) as judges, the information provided to the judge matters more than the specific judge model chosen. The study, titled "The Anatomy of a Blind Spot: What an LLM Judge Must Be Shown, and How Much," was conducted by Nitish Kumar et al.

The team varied only what the judge was shown across seven public benchmarks, comparing three judges with prices differing by 10- to 17-fold. Results showed that when a system revised its answers based on feedback from a judge denied the material it had to respect, it ended below its starting point on instruction following and question answering. This loss grew with each round and persisted even with a second judge family.

Additionally, the study found that on news summarization, a correct reference summary undid most of the source's contribution, whereas on single-answer questions, a reference helped. About a tenth of the source did almost all of its work. The authors committed 83 testable clauses before running; 50 went in the predicted direction, but most quantitative predictions from the theory failed. These results are currently first-party benchmarks (E2) and have not yet been independently reproduced.