독서리서치레이더투자 프레임워크
로그인 / 회원가입
← 읽기로 돌아가기
뉴스주제 · 决策模型选型之争중대2 분 소요DisconfirmAI 편집부

Decision Models Cannot Be Selected by Benchmark Scores: Microsoft Engineering VP Says 'Benchmaxed' Models Fail Rapidly in Enterprise Use

Executives from multiple decision model vendors point out that because these models are easily replicable and their real-world usage distribution differs significantly from benchmark tests, public scores cannot predict production performance. The true differentiation lies in whether a model can continuously improve judgment calibration using the customer's own data.

2026-11-18 02:10 GMT+8

Decision models are a new product category that emerged only in September 2026: they do not generate text but use the general intelligence of large language models to make quick judgments between preset options while outputting confidence levels. Data from Vercel’s AI model router shows that in October 2026, more than one-quarter of requests flowed to decision models rather than traditional large language models. One reason for this category's rapid growth is its low barrier to entry—TechCrunch reported that an Amazon engineer built a decision model over a single weekend, prompting major cloud providers to follow suit. On November 17, Microsoft CEO Satya Nadella announced that Microsoft-Decision-1.1 would be immediately rolled out across Azure.

As the number of models rapidly expanded, the selection problem became thorny. Achint Srivastava, Vice President of Engineering at Microsoft, wrote in an email: "What we see is that 'benchmaxed' models fall flat under real enterprise usage very quickly." He further pointed out that it is extremely challenging to deliver accuracy, reliability, speed, and cost efficiency within a single model; Microsoft-Decision-1.1 and TypeSafe’s Jev are among the few models that maintain strong performance across four dimensions even on "held-out" or unseen benchmarks.

Ushnish Sengupta, co-founder of UK-based AI startup Sqwish, offered another perspective: benchmark tests are more like "an invitation to try the model," while continuous tuning is the reason customers stay. Sqwish recently announced completing a $3 million pre-seed funding round. The differentiation of its decision model, "kettei fat," does not lie in initial benchmark scores but in built-in post-training tools—customers can continuously upload their own production data, allowing the model to constantly improve judgment calibration for specific tasks.

로그인 / 회원가입
독서리서치레이더투자 프레임워크

This logic aligns with the views of Diogo Almedia, CEO of TypeSafe: the most difficult part of building Jev was finely calibrating the model's judgment using statistically adjusted synthetic data. Sqwish also uses synthetic data but adds the capability for continuous fine-tuning with customer-owned data. Sqwish co-founder Federica Freddi positions this track as "System 1 intelligence"—working in concert with the "System 2 intelligence" of frontier large language models. She believes the market size for this layer is in the tens of billions of dollars, and for frontier labs like OpenAI, decision models are merely side projects.

출처

  • techcrunch.comTechCrunch↗
주제

决策模型选型之争

【背景】“决策模型”(Decision Model)作为一种不生成文本、仅对预设选项进行快速判断并输出置信度的AI架构,自2026年9月TypeSafe发布Jev以来迅速成为企业级AI的新宠。截至10月,Vercel AI路由中超过四分之一…

4 단계1 건업데이트됨 2026-11-20 14:18

현재까지의 과정

  1. 2026-09-15TypeSafe发布首个主流决策模型Jev,定义‘System One’智能范式,引发数十家公司跟进。github.com ↗
  2. 2026-10-27Hangten系统团队发布案例研究,显示决策模型作为护栏可将非前沿模型的幻觉率降低六倍,且延迟远低于LLM裁判。hangtensystem.com ↗
  3. 2026-11-06AI One Labs发表研究,证明经过4,998个合成样本微调的开源决策模型Clef,在企业工作流分类任务中准确率达96.56%,接近GPT-6 Luna水平。ai.one ↗

아직 해결되지 않은 문제

  • 在缺乏统一行业标准的情况下,企业如何量化评估决策模型在特定私有数据分布上的‘校准度’而非单纯准确率?
  • 随着OpenAI等前沿实验室进入该领域,专用决策模型是否会像早期LLM一样被整合进通用大模型的能力模块,从而失去独立存在的价值?
  • 对于高基数分类或复杂逻辑推理任务,现有决策模型(如Jev)的明显短板是否可通过混合架构(System One + System Two)完全弥补?
전체 주제 읽기 →
리서치 계속하기 →