ReadingResearchRadarInvestment framework
Sign in / Sign up
← Back to reading
Long-term reading2 min readDisconfirmAI Editorial

Mixup: Beyond Empirical Risk Minimization

Although large deep neural networks are powerful, they often exhibit undesirable behaviors such as memorizing training data and sensitivity to adversarial samples.

Verified 2026-11-21 00:02 GMT+8

Traditionally, these networks are trained by minimizing the average error on the training data, following the principle of Empirical Risk Minimization (ERM). However, when the number of model parameters is comparable to the number of training samples, ERM can lead models to "memorize" rather than "generalize" the data, resulting in unstable predictions on out-of-distribution samples.

To address this issue, the paper proposes the mixup method. This is a simple, data-agnostic data augmentation technique. Its core idea is to construct virtual training samples via convex combinations of two randomly selected training samples and their labels. Specifically, for input vectors $x_i$ and $x_j$, and corresponding one-hot encoded labels $y_i$ and $y_j$, new samples are generated as $\tilde{x} = \lambda x_i + (1-\lambda) x_j$ and $\tilde{y} = \lambda y_i + (1-\lambda) y_j$, where $\lambda$ is a mixing coefficient sampled from a Beta distribution. This approach implicitly encodes the prior knowledge that linear interpolation of feature vectors should result in linear interpolation of target values.

Experiments show that mixup significantly improves the generalization ability of state-of-the-art neural networks on datasets such as ImageNet-2012 and CIFAR-10. Additionally, it reduces the model's memorization of mislabeled data, increases robustness to adversarial samples, and stabilizes the training of Generative Adversarial Networks (GANs). The implementation of Mixup requires only a few lines of code with minimal computational overhead, making it an efficient and universal regularization technique. This 2017 paper, "Mixup: Beyond Empirical Risk Minimization," is suitable for researchers and engineers seeking to understand how simple data augmentation strategies can enhance the generalization and robustness of deep learning models. It is recommended to start reading from the introduction and methodology sections, focusing on how it replaces complex domain-specific data augmentations at extremely low computational cost.

Sign in / Sign up
ReadingResearchRadarInvestment framework

Sources

  • arxiv.org《mixup: Beyond Empirical Risk Minimization》(2017)↗
Continue researching →