Reading

In deep learning optimization, we often view momentum as merely an acceleration technique. However, Gabriel Goh points out in "Why Momentum Really Works" that this common understanding fails to explain many important behaviors of momentum. In fact, by studying convex quadratic models, we can understand the momentum mechanism more precisely.

Standard gradient descent is simple but performs poorly when facing ill-conditioned curvature. So-called ill-conditioned curvature refers to regions in the function landscape that are not properly scaled, such as narrow valleys or ravines. In these regions, iteration points either jump across the valley or approach the optimum with tiny steps, causing progress to stall in certain directions.

The momentum algorithm introduces short-term memory to gradient descent. Its core formulas are z^{k+1} = βz^k + ∇f(w^k) and w^{k+1} = w^k - αz^{k+1}. When β=0, it degenerates into ordinary gradient descent; but when β approaches 1 (e.g., 0.99), the algorithm achieves significant acceleration. This is not a simple trick, but a profound insight into the nature of optimization.

To understand this phenomenon, the author analyzes the convex quadratic function f(w) = 1/2 w^T A w - b^T w. Through eigenvalue decomposition of the symmetric matrix A, we decompose the error along each eigenvector direction. The error component in each direction evolves independently, with its convergence rate determined by (1-αλ_i)^k. This means there are huge differences in convergence speed across directions with different curvatures.

The condition number κ = λ_n / λ_1 directly measures the severity of ill-conditioned curvature. The larger κ is, the more significant the gap between the largest and smallest eigenvalues, leading to extremely slow gradient descent in certain directions. Momentum changes the dynamics of the iteration, allowing the algorithm to handle these high-condition-number problems more effectively, thereby theoretically providing up to a quadratic speedup.

For modern machine learning practitioners, understanding momentum is not just about knowing how to set hyperparameters, but also about understanding how optimizers navigate complex high-dimensional spaces. For example, suppose we are training a neural network with features having extreme scale differences; ordinary gradient descent may get stuck in local stagnation due to slow convergence in directions with small eigenvalues, whereas momentum can smooth oscillations and accelerate traversal through these "flat" regions by accumulating historical gradient information.

"Why Momentum Really Works" is a classic article published by Gabriel Goh in 2017. It is suitable for researchers and engineers who wish to deeply understand the underlying logic of optimization algorithms. It is recommended to start reading from the section on the analysis of convex quadratic models, which clearly demonstrates the connection between mathematical derivation and actual performance improvement.

At the height of the global financial crisis in March 2009, Jeremy Grantham of the US asset management firm GMO published the article "Reinvesting When Terrified." He observed that during market crashes, investors often fall into a psychological state known as "terminal paralysis." This state does not stem from a lack of funds or opportunities, but rather from the fear of making mistakes and the pressure of professional risk. As Grantham stated: "Otherwise, human nature is to become 'terminally paralyzed' in large-scale market sell-offs."

The article points out that cash becomes increasingly attractive in falling markets because each decline enhances the sense of security in holding cash. Those who appear smart and hold significant cash are reluctant to easily abandon this "smartness," while those who are over-invested are in a state of rigidity. The result is that "typically, those with large amounts of cash miss a significant portion of the market recovery." This phenomenon was evident in the market cycles of 1974 and 1999, where human nature under extreme pressure tends to maintain the status quo rather than take action.

To address this dilemma, Grantham's core solution is to formulate and strictly execute a "reinvestment battle plan." He emphasizes: "There is only one antidote for terminal paralysis: you must absolutely have a reinvestment battle plan and stick to it." Since any action requires overcoming paralysis, he suggests taking "a few big steps, rather than many small steps." While it may be tempting to buy all at once at the lowest point, buying in several large increments is safer without "signing a contract with the devil." Furthermore, pre-approval from the investment committee must be obtained before panic intensifies, as consensus will be difficult to reach once the "rigidity period" sets in.

Grantham also reminds readers that markets do not reverse when they see light at the end of the tunnel, but rather when everything looks pitch black, yet slightly less dark than the day before. This means waiting for clear positive signals often leads to missing the best entry timing. By pre-defining what conditions constitute "fully invested," investors can avoid emotional decision-making and achieve better risk-adjusted returns over the long term.

Recommended reading: Jeremy Grantham's "Reinvesting When Terrified" (2009), suitable for individual investors and institutional decision-makers looking to establish a disciplined reinvestment framework amidst severe market volatility.