New Study Proposes 'Accessibility' Metric: Large Model Benchmarks May Hide Capability Stability Gaps

A preprint published on arXiv titled "Revealing what benchmark scores hide: Jointly measuring the breadth and reliability of LLM capabilities using accessibility profiles" proposes a new evaluation metric called "Accessibility," aiming to address information loss in traditional large language model benchmarks that rely on single, fixed wordings.

The research team, led by Kazem Faghih, argues that the same underlying request can be expressed in many semantically equivalent ways, yet models often perform inconsistently across these variations. Traditional metrics like accuracy, best-case, or average-case scores fail to capture this distributional structure. "Accessibility" is defined as the probability that a model succeeds across meaning-preserving formulations.

Through experiments on six benchmarks and over twenty models, the study finds this tradeoff is common: for about half of the model pairs with nearly equal mean accessibility, which model appears better depends on whether retries are allowed. If one success out of five attempts suffices (Access@5), one model may win; if all five must succeed (Reliable@5), another may be superior. In some cases, the divergence between these two measures reaches up to 10 percentage points.

The research also reveals that scaling, reasoning, and prompt optimization can redistribute capability across inquiries, sometimes broadening the set of queries a model can handle while making those successes less repeatable. This finding transforms formulation sensitivity from a nuisance into a measurable property of capability, distinguishing what a model can sometimes do from what users can actually depend on.