"Recurrent Models of Visual Attention": Simulating Human "Saccades" with Reinforcement Learning to Break Through the Computational Bottleneck of Convolutional Networks
In the field of computer vision, although Convolutional Neural Networks (CNNs) have achieved great success, their computational costs are high.
In the field of computer vision, although Convolutional Neural Networks (CNNs) have achieved great success, their computational costs are high. As pointed out in the paper "Recurrent Models of Visual Attention," applying CNNs to large images causes the computation to grow linearly with the number of pixels, resulting in extremely long training and testing times. To address this issue, researchers at Google DeepMind, inspired by human visual perception, proposed a new recurrent neural network model.
When humans observe a scene, they do not process the entire image at once; instead, they selectively focus attention on specific parts of the visual space, gradually accumulating information to build an internal representation. This mechanism not only saves "bandwidth" but also reduces task complexity by ignoring irrelevant backgrounds (i.e., "clutter"). The core contribution of this paper is modeling visual attention as a sequential decision-making process. The model, known as the Recurrent Attention Model (RAM), interacts with the environment like an agent. At each step, it extracts local high-resolution information through a bandwidth-limited sensor (Glimpse Sensor) and determines the next fixation point based on previous gaze positions.
Since the process of selecting fixation points is non-differentiable, the research team employed reinforcement learning for end-to-end training. Specifically, backpropagation was used to train the neural network components, while Policy Gradient methods were utilized to solve the non-differentiability issues in control problems. Experiments showed that this model significantly outperformed CNN baselines on cluttered image classification tasks and could dynamically track objects without explicit supervision signals. This method demonstrates the potential of attention-based models in handling complex backgrounds and scaling to large input images.