LeituraPesquisaRadarFramework de investimento
Entrar / Cadastrar
← Voltar para a leitura
NotíciasTópico · JEVal决策基准Visão rápida1 min de leituraEditorial DisconfirmAI

Decision models lack reliability in long chains

New benchmark JEVal reveals that while general decision models are efficient for single-step judgments, their reliability drops in multi-step interactions due to error accumulation, and they exhibit systematic bias in social simulations.

2026-10-06 12:00 GMT+8

General decision models like Jev weaken when tasks require specialist knowledge or faithful uncertainty estimation, often overstating outcome probabilities.

In long-horizon, multi-step tests such as τ-bench, faster local decisions reduce median episode time but lower overall task success as errors accumulate over long trajectories.

Large-scale social simulations show these models approach generative LLMs in individual response prediction at lower cost, yet remain weaker in user profiling and exhibit larger aggregate estimation errors and systematic bias.

Fontes

  • arxiv.org↗
Tópico

JEVal决策基准

通用决策模型(如Jev)在单步判断中高效,但在多步交互中因误差累积导致可靠性下降。2026年10月,复旦大学团队发布JEVal基准,评估了25种模型配置,指出此类模型在长链任务和社会模拟中存在系统性偏差。尽管其推理成本极低且速度较快,但校准…

4 etapas1 peçasAtualizado 2026-10-07 15:21

Como chegamos aqui

  1. 2026-09-19
Entrar / Cadastrar
LeituraPesquisaRadarFramework de investimento
Benchmark Heaven运行JevBench v1.0,测试242个决策项,显示GPT-5.6 Luna与Jev 1.13.0准确率接近,但Jev成本更低。
benchmarkheaven.com ↗
  • 2026-09-20jeval工具对RAGTruth数据集进行基准测试,发现Jev在案例级判断中精度约82.8%,但存在大量弃权情况,校准误差较高。getjeval.com ↗
  • 2026-10-02arXiv提交论文《General Decision Models: Benchmarking and Insights Beyond Jev》,提出JEVal基准,涵盖11,257个实例,揭示决策模型在多步交互中的可靠性缺陷。arxiv.org ↗
  • O que ainda está indefinido

    • InnerJev-27B是否能在实际生产环境中持续保持0.1秒响应下的低校准误差?
    • 不同第三方基准(如JevBench与JEVal)对同一模型的评价差异源于测试集还是评分权重?
    • 用户能否接受高弃权率以换取高精度,还是更倾向于强制输出带来的潜在错误风险?
    Ler o tópico completo →
    Continuar pesquisando →