RLHF
Reinforcement learning from human feedback — humans rank model outputs and the model learns to prefer the higher-ranked ones. The main technique that made modern chat models actually work.
RLHF is a three-stage process. First, a base model is pre-trained on a huge text corpus (standard language-model training). Second, humans generate preference data by ranking multiple model outputs for the same prompt — which one is more helpful, more accurate, less harmful. Third, the model is fine-tuned using reinforcement learning to produce outputs that match the human preferences.
The technique bridged a specific gap. Base models trained purely on internet text tended to complete prompts in whatever style the corpus suggested (verbose, meandering, sometimes hostile, often unhelpful in a conversational sense). RLHF taught them to respond in the way humans actually want a chat assistant to respond.
Anthropic developed a related technique called Constitutional AI, where the model itself generates preference data by evaluating outputs against a written set of principles ("the constitution") rather than requiring humans to generate every ranking. Claude's training uses a combination of both approaches.
RLHF has known limitations: it can teach models to be sycophantic (agreeing with users rather than correcting them), it depends heavily on the quality and diversity of human raters, and it doesn't fully align model behavior with human values in edge cases. Successor techniques (RLAIF, DPO, and others) address specific weaknesses.