RLHF

Reinforcement Learning from Human Feedback (RLHF) optimizes a model using a reward signal derived from human preference judgments.

People rank answers; a reward model learns those preferences; the main model is nudged to match them.

RLHF typically trains a reward model on human preference comparisons, then optimizes a policy (the LLM) with reinforcement learning (e.g., PPO) or related preference optimization methods.

Preference optimization is a major reason chat models follow instructions more usefully than raw pretrained LMs.

• Assistant alignment
• Style and safety preference tuning
• Research on preference optimization alternatives

• Improves helpfulness and harmlessness tradeoffs (when data is good)
• Encodes nuanced human judgments

• Expensive preference data
• Reward hacking risks
• Preferences may not transfer across cultures or tasks

• RLHF is not the only alignment method
• It does not make models omniscient

Is RLHF the same as reinforcement learning?

RLHF is a specific use of RL (or preference optimization) where the reward comes from human preference models.

• InstructGPT paper (arXiv)

• Ouyang et al. — Training language models to follow instructions with human feedback