Continuous Control with Deep Reinforcement Learning: How the DDPG Algorithm Breaks Through Discrete Action Limits

In the field of artificial intelligence, enabling machines to directly learn complex physical control tasks from raw pixel input has always been a significant challenge. In 2015, a team from Google DeepMind in London, UK, led by Timothy P. Lillicrap and Jonathan J. Hunt, published a groundbreaking paper titled "Continuous control with deep reinforcement learning." This paper proposed the DDPG (Deep Deterministic Policy Gradient) algorithm, successfully extending the success concepts of Deep Q-Networks (DQN) to continuous action spaces.

Traditional DQN algorithms perform excellently when handling discrete actions (such as directional key presses), but are powerless when facing tasks that require continuous numerical outputs, such as robotic arm movements or vehicle steering. If one forcibly discretizes continuous actions, they encounter the "curse of dimensionality": for example, a 7-degree-of-freedom robotic arm, even with the coarsest discretization, would see its action space explode to 2,187 dimensions, making training difficult to converge and losing structural information between actions. The core innovation of DDPG lies in combining Deterministic Policy Gradient (DPG) with the stability techniques of DQN. It adopts an Actor-Critic architecture, where the "Actor" network directly outputs deterministic continuous actions, and the "Critic" network evaluates the value of those actions. To address instability caused by non-linear function approximation, DDPG introduces two key mechanisms: first, it uses a Replay Buffer, breaking data correlation through random sampling; second, it uses a Target Network to provide stable TD error calculation targets. Additionally, it applies batch normalization technology to further stabilize training.

Experimental results show that DDPG can robustly solve over 20 simulated physical tasks using the same hyperparameters and network structure, including inverted pendulum swinging, dexterous manipulation, bipedal walking, and car driving. More notably, in many tasks, DDPG can learn policies end-to-end directly from raw pixel inputs, with performance comparable to planning algorithms that possess complete dynamic model information. This achievement demonstrates the potential of deep reinforcement learning in handling high-dimensional continuous control problems, laying an important foundation for subsequent research in robotics and autonomous driving. For readers wishing to gain a deeper understanding of the application of reinforcement learning in continuous control, "Continuous control with deep reinforcement learning" (2015) is a must-read classic. It is recommended to start reading from the abstract and Section 3 on algorithm description, focusing on the policy gradient update rule in Equation (6) and the experimental setup in Section 4.