Reading

In 1992, Peter Dayan from the Centre for Cognitive Science at the University of Edinburgh, UK, and Christopher J. C. H. Watkins from London published the classic paper "Q-Learning" in the journal Machine Learning. This technical exposition aims to address the problem of how an agent can learn optimal action policies through trial and error in a controlled Markov domain.

The core contribution of this research lies in proposing and rigorously proving the convergence theorem for Q-Learning. The article states that Q-Learning is a simple method for agents to learn how to take optimal actions in a controlled Markov domain. It is essentially an incremental implementation of dynamic programming with limited computational requirements. Its working principle involves gradually improving the evaluation of the quality of specific actions in specific states. As long as all actions are repeatedly sampled across all states, and action values are represented discretely, Q-Learning converges to the optimal action values with probability 1. Additionally, the authors briefly discuss extensions involving non-discounted but absorbing Markov environments and cases where multiple Q-values can be updated per iteration.

For modern readers, understanding this mechanism helps grasp the foundational logic of deep reinforcement learning. For example, imagine a robot needing to find its way out of a maze; initially, it does not know which path is best. Through the Q-Learning algorithm, the robot updates its value estimates (Q-values) for the current state and action after each step based on whether it is approaching the exit (reward). After numerous repeated attempts, even without prior knowledge of the map, the robot gradually learns to select sequences of actions that yield the highest long-term returns, ultimately finding the shortest path. This process of accumulating global optimal strategies from local experiences constitutes the theoretical cornerstone established by Q-Learning.

Recommended reading: The original paper "Q-Learning" by Watkins and Dayan (1992). For AI researchers and students wishing to deeply understand the mathematical foundations of reinforcement learning, particularly convergence guarantees and their relationship with traditional dynamic programming, this is a must-read classic literature. It is suggested to start with the abstract and introduction, focusing on their definitions of "incremental methods" and "convergence conditions."

In his 1981 paper "Do Stock Prices Move Too Much to be Justified by Subsequent Changes in Dividends?", Robert Shiller challenged the Efficient Market Hypothesis. He constructed a valuation model arguing that actual stock prices should equal the rational expectation of the present value of future dividends. If this model holds, stock price volatility should be strictly bounded by changes in fundamentals.

To test this view, Shiller introduced the concept of "ex post rational prices" (p*), defined as theoretical values calculated based on dividends that actually occurred later. By comparing historical data of actual stock price indices (p) with p*, he found significant discrepancies between the two. These differences were not due to short-term information noise but represented structural deviations.

The core mechanism lies in the variance bounds test. According to efficient market theory, prediction errors should be uncorrelated with predictions, implying that the variance of actual prices must be less than or equal to the variance of ex post rational prices. However, data showed that the actual volatility of the S&P and Dow Jones indices far exceeded this theoretical upper limit. Shiller pointed out that such massive deviations indicate that markets do not always react rationally to information.

For example, the stock market crash from 1929 to 1932 could not be reasonably explained by dividend changes in subsequent years. Although corporate conditions were poor during the Great Depression, the decline in actual dividends was far insufficient to support such a drastic drop in stock prices. Shiller explicitly wrote in the text: "the stock market decline beginning in 1929 and ending in 1932 could not be rationalized in terms of subsequent dividends!"

Assuming current investors face similar severe market fluctuations, relying solely on the dividend discount model might lead them to underestimate risks or misjudge opportunities. Shiller's research suggests that when price volatility significantly deviates from fundamentals, there may be elements of overreaction or bubbles described by behavioral finance, rather than mere information updates. This provides an important theoretical framework for understanding market anomalies.

This classic paper is suitable for readers who wish to deeply understand market anomalies, the foundations of behavioral finance, and the limitations of asset pricing. It is recommended to start reading from the introduction, focusing on the data comparison in Figures 1 and 2 and the derivation process of the variance inequality. The work is titled "Do Stock Prices Move Too Much to be Justified by Subsequent Changes in Dividends?" and was published in 1981.