Principles of Momentum Optimization: Why It Overcomes Ill-Conditioned Curvature in Gradient Descent

In deep learning optimization, we often view momentum as merely an acceleration technique. However, Gabriel Goh points out in "Why Momentum Really Works" that this common understanding fails to explain many important behaviors of momentum. In fact, by studying convex quadratic models, we can understand the momentum mechanism more precisely.

Standard gradient descent is simple but performs poorly when facing ill-conditioned curvature. So-called ill-conditioned curvature refers to regions in the function landscape that are not properly scaled, such as narrow valleys or ravines. In these regions, iteration points either jump across the valley or approach the optimum with tiny steps, causing progress to stall in certain directions.

The momentum algorithm introduces short-term memory to gradient descent. Its core formulas are z^{k+1} = βz^k + ∇f(w^k) and w^{k+1} = w^k - αz^{k+1}. When β=0, it degenerates into ordinary gradient descent; but when β approaches 1 (e.g., 0.99), the algorithm achieves significant acceleration. This is not a simple trick, but a profound insight into the nature of optimization.

To understand this phenomenon, the author analyzes the convex quadratic function f(w) = 1/2 w^T A w - b^T w. Through eigenvalue decomposition of the symmetric matrix A, we decompose the error along each eigenvector direction. The error component in each direction evolves independently, with its convergence rate determined by (1-αλ_i)^k. This means there are huge differences in convergence speed across directions with different curvatures.

The condition number κ = λ_n / λ_1 directly measures the severity of ill-conditioned curvature. The larger κ is, the more significant the gap between the largest and smallest eigenvalues, leading to extremely slow gradient descent in certain directions. Momentum changes the dynamics of the iteration, allowing the algorithm to handle these high-condition-number problems more effectively, thereby theoretically providing up to a quadratic speedup.