Decision Models Cannot Be Selected by Benchmark Scores: Microsoft Engineering VP Says 'Benchmaxed' Models Fail Rapidly in Enterprise Use
Executives from multiple decision model vendors point out that because these models are easily replicable and their real-world usage distribution differs significantly from benchmark tests, public scores cannot predict production performance. The true differentiation lies in whether a model can continuously improve judgment calibration using the customer's own data.
Decision models are a new product category that emerged only in September 2026: they do not generate text but use the general intelligence of large language models to make quick judgments between preset options while outputting confidence levels. Data from Vercel’s AI model router shows that in October 2026, more than one-quarter of requests flowed to decision models rather than traditional large language models. One reason for this category's rapid growth is its low barrier to entry—TechCrunch reported that an Amazon engineer built a decision model over a single weekend, prompting major cloud providers to follow suit. On November 17, Microsoft CEO Satya Nadella announced that Microsoft-Decision-1.1 would be immediately rolled out across Azure.
As the number of models rapidly expanded, the selection problem became thorny. Achint Srivastava, Vice President of Engineering at Microsoft, wrote in an email: "What we see is that 'benchmaxed' models fall flat under real enterprise usage very quickly." He further pointed out that it is extremely challenging to deliver accuracy, reliability, speed, and cost efficiency within a single model; Microsoft-Decision-1.1 and TypeSafe’s Jev are among the few models that maintain strong performance across four dimensions even on "held-out" or unseen benchmarks.
Ushnish Sengupta, co-founder of UK-based AI startup Sqwish, offered another perspective: benchmark tests are more like "an invitation to try the model," while continuous tuning is the reason customers stay. Sqwish recently announced completing a $3 million pre-seed funding round. The differentiation of its decision model, "kettei fat," does not lie in initial benchmark scores but in built-in post-training tools—customers can continuously upload their own production data, allowing the model to constantly improve judgment calibration for specific tasks.