LectureRechercheRadarCadre d'investissement
Connexion / Inscription
← Retour à la lecture
ActualitésSujet · SFT与GRPO选型之争Matériel1 min de lectureRédaction DisconfirmAI

AWS experiment: choosing SFT or GRPO for fine-tuning depends on whether the target behavior already exists in the base model

The AWS machine learning team ran controlled experiments on two models (GPT-OSS-20B and Qwen2.5-7B) across two task types to validate a decision rule: SFT teaches a behavior the model completely lacks, while GRPO refines only a behavior the model can sometimes already produce. On a math task where the base model was already at peak accuracy, GRPO delivered no improvement.

2026-10-21 01:48 GMT+8

GPT-OSS-20B is an open-weight reasoning model with a stubborn habit: regardless of which language the user asks in, its chain-of-thought almost always generates in English. The AWS machine learning team found that on a 50-example multilingual reasoning test, the base model used the correct target language only 22 percent of the time, and that 22 percent came almost entirely from the English samples—French, Spanish, and Italian scored zero, with German at just 10 percent.

The team then ran supervised fine-tuning (SFT) for 1,000 steps using a LoRA adapter (r=8, α=16) across four ml.g6e.12xlarge instances. The result was direct: reasoning-language accuracy jumped from 22 percent to 100 percent. But answer-language reached only 62 percent—the model learned to think in French yet often switched back to English for the final answer.

This is precisely where GRPO (Group Relative Policy Optimization) applies. GRPO works by sampling a group of responses per prompt and pushing the policy toward the above-average ones. If the target behavior has zero probability in the base distribution, there is almost nothing for GRPO to reinforce. After SFT, the model could already reason in the correct language 100 percent of the time and occasionally produce the answer in the right language as well, giving GRPO a signal to reinforce.

Connexion / Inscription
LectureRechercheRadarCadre d'investissement

Sources

  • aws.amazon.com↗
Sujet

SFT与GRPO选型之争

【背景】在大模型微调中,选择监督微调(SFT)还是组相对策略优化(GRPO)常令开发者困惑。近期多项独立研究——包括美国亚马逊云科技(AWS)、加拿大Dialpad公司及瑞士Idiap研究所——通过对照实验揭示了一条核心原则:SFT用于教授…

4 étapes1 piècesMis à jour 2026-10-21 15:55

Comment on en est arrivé là

  1. 2025-02-17Trelis Research发布分析指出,对于数学推理任务,若基座模型已有一定解题能力,SFT配合拒绝采样往往比直接GRPO更稳健,GRPO可能因格式奖励导致准确率下降。trelis.substack.com ↗
  2. 2026-07-09Idiap研究所等机构发表论文《When Synthetic Speech Is All You Have》,证明在合成语音ASR适配中,GRPO相比SFT能将词错误率(WER)降低40%,因为RL优化序列级指标而非逐token交叉熵。arxiv.org ↗
  3. 2026-09-17Dialpad公司发表受控研究,对比Qwen3系列模型的工具调用能力。结果显示SFT+LoRA在18个分布内设置中15次胜出,GRPO仅在跨数据集迁移时略优,且平均优势小于1个百分点。arxiv.org ↗

Ce qui reste non résolu

  • 在何种具体的奖励函数设计下,GRPO能在SFT之后产生显著的叠加效应而非边际递减?
  • 对于资源受限的边缘设备部署,GRPO的高计算开销是否使其在实际生产中不可行?
  • 是否存在统一的量化指标来预先判断某项任务属于‘能力空白’(需SFT)还是‘能力波动’(需GRPO)?
Lire le sujet complet →
Continuer la recherche →