Reinforcement Learning vs Supervised Fine-Tuning in LLM Post-Training

This article explains the two pillars of post-training for large language models (LLMs): reinforcement learning (RL) and supervised fine-tuning (SFT). Reinforcement learning uses prompt-to-response generation, then applies feedback as a reward signal to update model behavior. The reward can come from verifiers (fast checks like code compilation, math correctness, factuality), human feedback via RLHF using a trained reward model, or “LLM-as-judge”/RLAIF where an AI judge scores outputs using rubrics to reduce single-score gaming. It also covers major RL update methods: REINFORCE, PPO (with a critic and clipped updates for stability), DPO (preference-optimization that avoids an explicit RL loop), and GRPO (group-relative optimization that removes the critic by normalizing rewards within a response set). The article notes RL post-training is usually less stable than supervised learning and often relies on a KL penalty to prevent drift from the base model. Supervised fine-tuning (SFT) teaches by demonstration: train on curated {prompt, ideal response} pairs using next-token prediction over response tokens only. SFT can use human-written demonstrations, synthetic data (e.g., Alpaca-style approaches), curated data transformed from existing human content, or rejection sampling where many generated candidates are filtered by reward/verifiers/judges to build better training sets. It argues both methods have limits: RL targets usability via feedback but can be noisy; SFT is simpler but bounded by demonstration coverage and can “mode average” conflicting styles. It highlights why frontier models often combine RL and SFT, citing DeepSeek R1-Zero as an example where RL improved reasoning but still required usability fixes.