Researchers Propose Bellman Language Model: Pre-solving MDPs to Optimize AI Agents

Researchers Adarsh Vatsa and William Eiers submitted a paper on October 17, 2026, proposing a new framework called "Bellman Language Models" (BLM) to address the inefficiency of Large Language Model (LLM) agents in repetitive tasks. Traditional LLM agents often re-derive decisions on every turn, even when facing states they have encountered before. The core idea of BLM is "Discretize, Then Decide": it converts the agent's interaction history into a discrete Markov Decision Process (MDP), pre-computes the optimal policy for each state, and then uses a lightweight controller to execute these fixed policies, calling the LLM only for unseen states.

The abstract notes that in bounded agentic workflows, the decision space tends to saturate as agents repeatedly meet the same states. By treating these decisions as an optimization problem solved once, costs can be significantly reduced and reliability improved. Experimental data shows that on the tau2-bench telecom benchmark, the controller alone solves 79% of tasks without using the model. When running as a BLM, the open-source model Qwen3.8-27B improves its task resolution rate from 29% to 96%, while reducing model calls by 93%. For strong models like Claude Opus 5.5, the BLM mode cuts model calls and cost per task by about 98% and allows it to solve two additional tasks.

However, the method is not universal. The authors explicitly acknowledge that where the hard part is writing content rather than choosing moves, such as resolving complex issues in SymPy, few decisions can be solved in advance, and BLM helps little. This indicates the framework is particularly suitable for industrial-grade agent scenarios with clear state transition logic and high repetition but templated content generation, such as customer service automation or inventory management.