Q-Learning: Proof of Convergence in Reinforcement Learning and Incremental Dynamic Programming Mechanisms

In 1992, Peter Dayan from the Centre for Cognitive Science at the University of Edinburgh, UK, and Christopher J. C. H. Watkins from London published the classic paper "Q-Learning" in the journal Machine Learning. This technical exposition aims to address the problem of how an agent can learn optimal action policies through trial and error in a controlled Markov domain.

The core contribution of this research lies in proposing and rigorously proving the convergence theorem for Q-Learning. The article states that Q-Learning is a simple method for agents to learn how to take optimal actions in a controlled Markov domain. It is essentially an incremental implementation of dynamic programming with limited computational requirements. Its working principle involves gradually improving the evaluation of the quality of specific actions in specific states. As long as all actions are repeatedly sampled across all states, and action values are represented discretely, Q-Learning converges to the optimal action values with probability 1. Additionally, the authors briefly discuss extensions involving non-discounted but absorbing Markov environments and cases where multiple Q-values can be updated per iteration.

For modern readers, understanding this mechanism helps grasp the foundational logic of deep reinforcement learning. For example, imagine a robot needing to find its way out of a maze; initially, it does not know which path is best. Through the Q-Learning algorithm, the robot updates its value estimates (Q-values) for the current state and action after each step based on whether it is approaching the exit (reward). After numerous repeated attempts, even without prior knowledge of the map, the robot gradually learns to select sequences of actions that yield the highest long-term returns, ultimately finding the shortest path. This process of accumulating global optimal strategies from local experiences constitutes the theoretical cornerstone established by Q-Learning.