Study: Non-English Languages Consume Up to 5x More AI Tokens, Creating a 'Tokenizer Tax'
A study reported by Rest of World finds significant token cost disparities across languages, disadvantaging non-English users.
According to a report by Rest of World published on October 26, 2026, a study conducted by the Coalition for Language Equity in AI Research reveals that different languages consume vastly different amounts of tokens when interacting with large language models, creating what is termed a "tokenizer tax."
The research indicates that among the top 20 most popular languages on the internet, the most expensive ones use between 1.5 and 5 times more tokens than the cheapest. When the sample expanded to 200 languages, the gap reached up to 17 times in certain models. Languages using non-Latin scripts, such as Burmese, Tibetan, Tifinagh, Khmer, Lao, and Ol Chiki, consistently faced disadvantages across nearly all tested models.
This disparity affects more than just billing; it increases latency and reduces the effective context window available to users. For instance, the same sentence input in Malayalam may be split into more fragments than in English, consuming more computational resources. While previous academic work has noted similar phenomena, this study provides broader cross-language quantification, highlighting structural inequalities in AI infrastructure.