StatQuest with Josh Starmer
18 minutes
The InstructGPT recipe behind aligned chat models: supervised fine-tuning, reward-model training on human preference pairs, and PPO with a KL penalty against the reference model.
Why this Concept is marked deprecated
Superseded for routine preference alignment by direct preference optimization, which drops the separate reward model and the online RL loop for a single classification loss.