Highway Networks: How Gating Mechanisms Break Through Deep Neural Network Training Bottlenecks

In the early days of deep learning, increasing network depth often led to training difficulties. Although theoretical proofs showed that deep networks could represent complex functions more efficiently, in actual optimization, gradient signals decayed or exploded as depth increased, making it difficult to train networks with hundreds of layers directly using standard stochastic gradient descent.

Highway Networks, proposed by Srivastava et al., aimed to alleviate this issue. The core idea was to introduce learnable gating units, allowing information to flow unimpeded between network layers. This architecture was inspired by Long Short-Term Memory (LSTM) networks; by regulating information flow, it enabled the network to smoothly switch between "transforming" and "preserving" inputs.

Specifically, a highway layer defines two nonlinear transformations: the transform gate T and the carry gate C. The output y consists of two parts: a weighted combination of the transformed H(x) and the original input x. The formula is y = H(x)·T(x) + x·(1-T(x)). When T approaches 1, the network performs transformation; when T approaches 0, information passes through directly, forming an "information highway."

Experiments demonstrated that this architecture allowed highway networks with hundreds of layers to be optimized directly using SGD. Unlike traditional networks, the optimization performance of highway networks was almost unaffected by depth. This opened up possibilities for researching extremely deep and efficient architectures, achieving test accuracy similar to that on CIFAR-10 without relying on pre-trained teacher networks.

For today's readers, understanding this mechanism helps grasp the predecessor logic of residual connections (ResNet). Suppose we are building an extremely deep feature extractor and find severe vanishing gradients; referring to the gating idea of Highway Networks—explicitly controlling the proportion of preserved information flow rather than relying entirely on weight initialization—may provide another debugging perspective.