Feature Visualization: How Neural Networks Build Image Understanding
'Feature Visualization' by Chris Olah, Alexander Mordvintsev, and Ludwig Schubert, published in 2017, is a classic in the field of interpretable AI. The core idea lies in generating examples through 'optimization' rather than simple data retrieval, revealing what internal neurons in neural networks are actually searching for. The authors note that while the concept is simple, making the optimization process work effectively involves many nuances, such as selecting the right optimization objective. For classifiers, directly optimizing the softmaxed probabilities often performs poorly because the model can boost the target class's probability by suppressing evidence for other classes; optimizing the logits before softmax typically produces images of higher visual quality.
Write-upStandard
To address the issue that a single optimization result may only show one aspect of a feature, the paper introduces diversity techniques. A simple and effective method is to add a 'diversity term' to the objective function, using the Gram matrix from style transfer to compute negative cosine similarity between different visualizations, forcing the generated images to be distinct. This approach reveals how low-level neurons respond differently to textures, or how high-level neurons recognize commonalities across different objects (such as various types of balls). However, it also exposes that neurons are not perfect semantic units—sometimes they mix responses to unrelated concepts (e.g., animal faces and car bodies).
The paper further proposes a geometric perspective on neuron interactions, defining the 'activation space' as the set of all possible combinations of neuron activations. In this framework, individual neurons are basis vectors, and combinations of neurons are vectors within the space. Research shows that random directions in this space often also yield interpretable results, although less frequently than basis vector directions. This suggests that understanding the network should not be limited to individual neurons but should focus on the overall representational structure within the high-dimensional space. For researchers and engineers seeking to deeply understand the black-box mechanisms of deep learning, this 2017 paper provides crucial insights from theory to practice, and it is recommended to begin with the section 'Feature Visualization by Optimization' to grasp the foundational methodology.