AWS experiment: choosing SFT or GRPO for fine-tuning depends on whether the target behavior already exists in the base model
The AWS machine learning team ran controlled experiments on two models (GPT-OSS-20B and Qwen2.5-7B) across two task types to validate a decision rule: SFT teaches a behavior the model completely lacks, while GRPO refines only a behavior the model can sometimes already produce. On a math task where the base model was already at peak accuracy, GRPO delivered no improvement.
ImportanceMaterialEvidenceE3 inspectableWrite-upStandard
GPT-OSS-20B is an open-weight reasoning model with a stubborn habit: regardless of which language the user asks in, its chain-of-thought almost always generates in English. The AWS machine learning team found that on a 50-example multilingual reasoning test, the base model used the correct target language only 22 percent of the time, and that 22 percent came almost entirely from the English samples—French, Spanish, and Italian scored zero, with German at just 10 percent.
The team then ran supervised fine-tuning (SFT) for 1,000 steps using a LoRA adapter (r=8, α=16) across four ml.g6e.12xlarge instances. The result was direct: reasoning-language accuracy jumped from 22 percent to 100 percent. But answer-language reached only 62 percent—the model learned to think in French yet often switched back to English for the final answer.
This is precisely where GRPO (Group Relative Policy Optimization) applies. GRPO works by sampling a group of responses per prompt and pushing the policy toward the above-average ones. If the target behavior has zero probability in the base distribution, there is almost nothing for GRPO to reinforce. After SFT, the model could already reason in the correct language 100 percent of the time and occasionally produce the answer in the right language as well, giving GRPO a signal to reinforce.