Reinforcement learning from human feedback (RLHF): RLHF is a training step in which human raters compare model responses, their preferences train a reward model, and the language model is then tuned to produce answers people rate highly. It is how raw text predictors became helpful assistants.
A freshly pre-trained language model is good at continuing text but not at following instructions or staying polite. RLHF fixes that in three steps. First, people write examples of good responses. Second, raters rank several model answers to the same prompt from best to worst, and those rankings train a separate reward model that predicts what people prefer. Third, the language model is optimized with reinforcement learning to score well on that reward model.
OpenAI described this recipe in its 2022 InstructGPT paper, and it became the standard way to turn a base model into a chat assistant. Since then, labs have added variants: direct preference optimization skips the separate reward model, and Anthropic's constitutional AI uses a written set of principles plus AI-generated feedback to reduce the amount of human labeling.
RLHF shapes personality as much as capability. The tone, the hedging, the refusals, and the tendency to agree with you all come partly from what raters rewarded. It also introduces known side effects: sycophancy, over-cautious refusals, and answers that sound confident because confident answers got rated higher.
Example at work
A paralegal asks an assistant whether a filing deadline was missed and gets a reassuring "you should be fine." Knowing that assistants are tuned to please, she rephrases: "Assume I am wrong and argue that the deadline was missed, citing the rule." The second answer surfaces a weekend rollover rule she had overlooked, which she then confirms in the court's own rules.
Why it matters
Understanding RLHF tells you why an assistant flatters your ideas and why asking it to argue the other side often produces a more useful answer. It also explains why two models with similar underlying knowledge can feel completely different to work with.
Related terms
- Reinforcement learningReinforcement learning is a way of training AI by trial and error: the system takes actions, receives rewards or penalties based on the outcomes, and adjusts its behavior to earn more reward over time.
- AlignmentAlignment is the work of making an AI system pursue the goals and values its designers and users actually intend, rather than a shortcut, a literal reading of the instructions, or a goal of its own.
- Fine-tuningFine-tuning is extra training that adjusts an existing AI model's weights using a smaller, targeted set of examples so it performs better on a specific task, style, or domain. It changes the model itself, unlike prompting, which only changes the instructions.
- Pre-trainingPre-training is the first and largest phase of building an AI model, in which it learns general patterns from a massive dataset, such as predicting the next word across trillions of words of text. Later phases then shape it into a useful assistant.
- HallucinationA hallucination is when an AI model produces information that is false or fabricated but presents it confidently, such as an invented statistic, a citation to a paper that does not exist, or a made-up product feature.