ReadingResearchRadarInvestment framework
Sign in / Sign up
← Back to reading
Long-term reading1 min readDisconfirmAI Editorial

"Asynchronous Methods for Deep Reinforcement Learning": Replacing Experience Replay with Parallel Agents for Efficient Training on a Single CPU

In 2016, the Google DeepMind team published the paper "Asynchronous Methods for Deep Reinforcement Learning" (A3C)...

Verified 2026-11-18 00:04 GMT+8

In 2016, the Google DeepMind team published the paper "Asynchronous Methods for Deep Reinforcement Learning", introducing the A3C algorithm. Prior to this, deep reinforcement learning primarily relied on an "experience replay" mechanism to stabilize training, which involved storing historical agent data and randomly sampling updates. While effective, this approach required significant memory and computational resources and restricted the use to off-policy algorithms.

The core innovation of this paper was abandoning experience replay in favor of an asynchronous parallel architecture. By running multiple independent "actor-learner" threads on a standard multi-core CPU, each thread interacted with the environment and computed gradients independently. This parallelism naturally decoupled the temporal correlation of data, making the training process more stable while enabling the use of a broader range of online policy algorithms (such as Sarsa and n-step methods).

Experimental results showed that the best-performing Asynchronous Advantage Actor-Critic (A3C) method surpassed state-of-the-art approaches in Atari game domains. Notably, it achieved this using only a single multi-core CPU rather than a GPU, halving training time. This finding demonstrated that efficient deep reinforcement learning could be achieved without expensive hardware or distributed clusters, providing a lightweight design philosophy for subsequent algorithms.

Sources

  • arxiv.org《Asynchronous Methods for Deep Reinforcement Learning》(2016)↗
Sign in / Sign up
ReadingResearchRadarInvestment framework
Continue researching →