AnthropicBased on real interviews
RLHF failure modes
Explain how human preference training can produce behaviour that misses the intended objective.
Unlock with Full AccessThe challenge
A research lead asks you to identify specific failures in RLHF and decide which deserves further investigation. Follow the failure through the preference signal, reward model or policy optimisation step. Your recommendation needs evidence, a mitigation within the RLHF pipeline and an experiment capable of challenging your explanation.
What you'll practise
Preference and reward signals
Connect the observed behaviour to a specific part of the RLHF pipeline.
Evaluation design
Distinguish competing explanations rather than simply collecting examples of failure.
Research judgement
Choose a useful next experiment and acknowledge the limits of the proposed fix.
Your recommendation
Write your reasoning and submit it for feedback.
The full brief and feedback are included with Full Access.