LecturaInvestigaciónRadarMarco de inversión
Iniciar sesión / Registrarse
← Volver a la lectura
NoticiasTema · 大模型可及性指标Material2 min de lecturaEditorial de DisconfirmAI

New Study Proposes 'Accessibility' Metric: Large Model Benchmarks May Hide Capability Stability Gaps

An arXiv preprint introduces 'Accessibility', showing that models with identical average accuracy can differ significantly in stability and breadth when handling synonymous prompts, challenging traditional fixed-wording evaluation paradigms.

2026-10-20 12:00 GMT+8

A preprint published on arXiv titled "Revealing what benchmark scores hide: Jointly measuring the breadth and reliability of LLM capabilities using accessibility profiles" proposes a new evaluation metric called "Accessibility," aiming to address information loss in traditional large language model benchmarks that rely on single, fixed wordings.

The research team, led by Kazem Faghih, argues that the same underlying request can be expressed in many semantically equivalent ways, yet models often perform inconsistently across these variations. Traditional metrics like accuracy, best-case, or average-case scores fail to capture this distributional structure. "Accessibility" is defined as the probability that a model succeeds across meaning-preserving formulations.

Through experiments on six benchmarks and over twenty models, the study finds this tradeoff is common: for about half of the model pairs with nearly equal mean accessibility, which model appears better depends on whether retries are allowed. If one success out of five attempts suffices (Access@5), one model may win; if all five must succeed (Reliable@5), another may be superior. In some cases, the divergence between these two measures reaches up to 10 percentage points.

The research also reveals that scaling, reasoning, and prompt optimization can redistribute capability across inquiries, sometimes broadening the set of queries a model can handle while making those successes less repeatable. This finding transforms formulation sensitivity from a nuisance into a measurable property of capability, distinguishing what a model can sometimes do from what users can actually depend on.

Iniciar sesión / Registrarse
LecturaInvestigaciónRadarMarco de inversión

Fuentes

  • arxiv.org↗
Tema

大模型可及性指标

马里兰大学计算机系研究团队于2026年10月发布预印本论文,提出“可及性”(Accessibility)这一新指标,旨在揭示传统大模型基准测试中单一固定措辞所掩盖的能力稳定性差异。该研究指出,两个平均准确率相同的模型,在应对不同语义等价提问…

2 pasos1 piezasActualizado 2026-11-07 19:14

Cómo llegó hasta aquí

  1. 2026-10-18Kazem Faghih等来自马里兰大学的作者提交arXiv预印本《Revealing what benchmark scores hide》,正式提出“可及性”概念,定义为模型在保持语义不变的多种表述下成功响应的概率。arxiv.org ↗
  2. 2026-10-20媒体跟进报道该研究,强调其挑战了传统单一固定措辞的评估范式,指出平均性能相同的模型在实际部署中因“广度”与“可靠性”权衡不同而适用场景迥异。arxiv.org ↗

Lo que aún está pendiente

  • “可及性”指标是否会被主流基准测试平台(如Hugging Face Open LLM Leaderboard)采纳为标准评估维度?
  • 在实际工程应用中,如何量化“重试成本”以决定是优先选择高Access@5还是高Reliable@5的模型?
  • 该研究提出的代码库(github.com/kazemf78/access-eval)能否被第三方复现并验证其在更大规模模型上的普适性?
Leer el tema completo →
Continuar investigando →