Apple Study: Bilingual Data Mixing Outperforms Hyperparameter Tuning in Data-Constrained Pre-training

In data-constrained language model pre-training, mixing in data from a high-resource auxiliary language yields larger improvements than aggressive hyperparameter tuning.

For most low-resource languages, models must repeat training data many times, which degrades generalization. Traditional remedies include regularizing by shrinking weights (high weight decay) or expanding the training distribution by mixing auxiliary data.

The Apple ML Research team compared these levers across four model scales (150M to 1.43B parameters) using Arabic as the target and English as the auxiliary over approximately 1,000 runs. They found that mixing boosts performance by an amount equivalent to 2–3× the unique target data on validation loss and 2–13× on downstream task accuracy. The study also reveals that target-language validation loss systematically underestimates mixing's value because it captures only regularization effects, not the new knowledge contributed.

This conclusion is based on first-party benchmarks (E2) and has not yet been independently reproduced. The authors recommend prioritizing the mixing ratio over hyperparameter tuning and transferring hyperparameters from small proxy models via μP.