DARTS: Differentiable Architecture Search via Continuous Relaxation
In the field of Neural Architecture Search (NAS), traditional methods such as reinforcement learning or evolutionary algorithms typically treat the search space as a discrete and non-differentiable black-box problem, resulting in extremely high computational costs.
In the field of Neural Architecture Search (NAS), traditional methods such as reinforcement learning or evolutionary algorithms typically treat the search space as a discrete and non-differentiable black-box problem, resulting in extremely high computational costs. For example, obtaining state-of-the-art architectures on CIFAR-10 and ImageNet previously required thousands of GPU days. DARTS (Differentiable Architecture Search) proposes a new paradigm based on continuous relaxation, transforming discrete architecture selection into continuous variables, thereby allowing efficient optimization using gradient descent.
The core of this method lies in constructing a mixed cell containing candidate operations. The edges between each node are no longer single operations but softmax-weighted combinations of all possible operations. Through bilevel optimization, the model simultaneously learns network weights and architecture parameters (i.e., operation mixing weights). This design not only simplifies the search process but also enables the algorithm to handle both convolutional and recurrent architecture types. Experiments show that DARTS achieves a 2.76% test error rate with 3.3M parameters on CIFAR-10, performing comparably to regularized evolutionary methods that use three orders of magnitude more computational resources, while also surpassing existing automatic search baselines on the Penn Treebank language modeling task.
For researchers and engineers seeking to deeply understand the underlying mechanisms of automated machine learning (AutoML), the paper "DARTS: Differentiable Architecture Search" (published in 2018) is a must-read classic. It demonstrates how to transform discrete combinatorial optimization problems into differentiable ones, laying the theoretical foundation for subsequent efficient architecture searches. It is recommended to start reading from Section 2 "Differentiable Architecture Search" of the paper, focusing on the derivation of the continuous relaxation formula and bilevel optimization.