AnthropicBased on real interviews
Research memo: diagnosing a mid-run eval decline
Training loss keeps falling, but evaluation results are getting worse. Decide what to investigate and what to do with the run.
Unlock with Full AccessThe challenge
A supervised fine-tuning run is approaching its release deadline when the target evaluation starts declining. Use the available checkpoints, curves and sample generations to develop an investigation plan. Your memo needs to explain what you would check first and how the findings would affect the run.
What you'll practise
Competing explanations
Reason about why training loss and evaluation results might diverge.
Experiment design
Choose checks that help distinguish between your hypotheses.
Decisions under a deadline
Prioritise the investigation within the remaining time and compute.
Your investigation plan
Explain what you would check and how it affects the run.
The full brief and feedback are included with Full Access.