"ImageNet Classification with Deep Convolutional Neural Networks": How ReLU and Multi-GPU Parallelism Enabled AlexNet

In 2012, the paper "ImageNet Classification with Deep Convolutional Neural Networks" (commonly known as AlexNet), co-authored by Alex Krizhevsky, Ilya Sutskever, and Geoffrey E. Hinton of the University of Toronto (Canada), marked a turning point for deep learning in the field of computer vision. The core contribution of this paper was to demonstrate that through specific architectural optimizations and hardware utilization, deep convolutional neural networks could achieve performance far surpassing traditional methods on large-scale image datasets.

The paper first addressed the issue of training efficiency. Traditional saturated non-linear activation functions (such as tanh or sigmoid) were slower during gradient descent training. The authors adopted unsaturated neurons called Rectified Linear Units (ReLUs), defined as f(x) = max(0, x). The text notes, "Deep convolutional neural networks with ReLUs train several times faster than their equivalents with tanh units." This improvement made it possible to train large networks with 60 million parameters within a limited time frame.

Secondly, to address the problem of model size exceeding the memory limits of a single GPU, the authors implemented a multi-GPU parallel training strategy. Since a single GTX 580 GPU had only 3GB of memory, and the 1.2 million training images were sufficient to train a network too large to fit on a single card, they stated, "we spread the net across two GPUs." This parallelization scheme not only broke through hardware bottlenecks but also improved computational efficiency by reducing communication overhead, ultimately lowering top-1 and top-5 error rates by 1.7% and 1.2%, respectively.

Additionally, the paper mentioned the Dropout regularization method to reduce overfitting in fully connected layers, as well as Local Response Normalization to assist generalization. These combined techniques enabled AlexNet to achieve a top-5 test error rate of 15.3% in the ILSVRC-2012 competition, significantly leading the runner-up's 26.2%.