Definition
A method that learns a reward signal from human evaluations or preferences, then uses reinforcement learning to adjust model behavior.
Preference-learning research in 2017 learned rewards through human comparisons. Language-model research in 2022 combined supervised demonstrations, preference rankings, reward models and reinforcement learning to improve instruction following.
RLHF is one approach to post-training; not all preference optimization or instruction fine-tuning should be called RLHF. Human preferences do not always align with factual correctness, long-term utility or safety objectives.
Principles and impact
Problems solved
- Introduced human judgment into tasks for which reward functions are difficult to write directly.
Principles
- Learns a reward model corresponding to preferences, then optimizes behavior using that signal.
Improvements
Not recorded
Significance
- Incorporated human evaluations into the process of adjusting model objectives.
History and applications
People
Paul F. Christiano、Long Ouyang
Organizations
Not recorded
Events
Not recorded
Applications
Learning behavioral preferences、Language-model post-training
Limitations
- Reward-model errors, preference biases and overoptimization of rewards can cause behavior to deviate from the intended goals.
Upstream and downstream relations
Upstream
Downstream
Sources: Training Language Models to Follow Instructions with Human Feedback