FLAN Paper: How Instruction Fine-Tuning Enables Zero-Shot Learning in Large Language Models

In 2021, the Google Research team published "Finetuned Language Models Are Zero-Shot Learners" (the FLAN paper), proposing a simple method to significantly improve the zero-shot learning capability of language models through "instruction fine-tuning." The core of this research lies in converting natural language processing tasks into formats with explicit instructions, such as "Is the sentiment of this sentence positive or negative?" or "Translate 'Hello' into Chinese," and fine-tuning the model on more than 60 such datasets.

Traditional large-scale pre-trained models (such as GPT-3) perform well under few-shot prompting but often struggle to understand tasks that differ significantly from pre-training data formats in zero-shot scenarios. The FLAN paper points out that by teaching models to follow instructions through supervised fine-tuning, they can generalize to unseen task types. Researchers grouped NLP datasets by task type; during training, one group of tasks was excluded, and the model's zero-shot performance on that group was evaluated only at the testing stage. Experimental results showed that the 137B-parameter FLAN model, after instruction fine-tuning, surpassed the 175B-parameter GPT-3 zero-shot performance on 20 out of 25 unseen tasks, and significantly outperformed GPT-3's few-shot performance on some reasoning and reading comprehension tasks.

This finding reveals three key factors important for the success of instruction fine-tuning: the quantity of the fine-tuning dataset, the model size, and the natural language instructions themselves. For current AI developers, FLAN demonstrates a path to stimulate the latent capabilities of models through structured instructions without requiring massive amounts of labeled data. Readers are advised to start with the abstract and introduction of the paper, focusing on the methodology comparison shown in Figures 1 and 2, to understand how instruction fine-tuning combines the advantages of the pre-training-fine-tuning paradigm and the prompting paradigm. This paper, published in 2021, titled "Finetuned Language Models Are Zero-Shot Learners," laid an important foundation for the instruction-following capabilities of subsequent large models.