Stanford Study: 44% of Widely Cited LLM Claims Expired, Average Lifespan Just 15.7 Months
Stanford researchers found that nearly half of widely cited LLM claims have expired in current models, with capability limitations fading fastest.
According to a report by technology media outlet The Decoder, Stanford University researchers Owen Queen and James Zou tested 50 widely cited claims about large language models (LLMs) against 20 current mainstream models. The results showed that 44% of the claims had already "expired," while another 34% had significantly weakened. Only 16% still held true. Among the 22 findings with an estimated expiration date, the average lifespan was just 15.7 months.
The study found that claims about what models "can't do" expire the fastest. Previously widely discussed weaknesses, such as counting letters in "strawberry," arithmetic in unfamiliar number systems, and losing information in the middle of long contexts, have largely disappeared in current models. In contrast, claims involving model preferences and behavioral tendencies prove more durable. Features like sycophancy, political bias, and self-preference when acting as evaluators remain significant across multiple models.
The research emphasizes that as AI development accelerates, empirical findings about LLMs should be presented as observations about specific models at a specific point in time, rather than as general truths. This conclusion reminds readers and researchers that many classic conclusions about AI capabilities from the past may no longer apply to current technical realities.