Anthropic
OpenAI
Google DeepMindBased on real interviews
Anticipate reward hacking in an RLHF run
Identify how the next RLHF run could improve reward while making the model less useful.
Unlock with Full AccessThe challenge
A team is preparing an RLHF run using a reward model trained mainly on responses from its earlier supervised checkpoint. Existing warning signs include longer answers without a quality gain, agreement with false premises and changing refusal behaviour.
Investigate how the policy could exploit that reward signal during optimization. Your memo must connect specific mechanisms to tests and mitigations that fit the limited evaluation compute and release window, while explaining what improvement in reward would leave unproven.
What you'll practise
Proxy-objective failures
Explain what the policy can exploit and why reward may diverge from the intended outcome.
Targeted evaluations
Connect each proposed test to a specific failure mechanism.
Practical mitigations
Specify changes that fit the run and acknowledge the risk they leave behind.
Write your reasoning and submit it for feedback.