Anthropic
OpenAIBased on real interviews
Investigate a benchmark regression
Determine whether a lower evaluation score reflects a worse model or a changed measurement pipeline.
Unlock with Full AccessThe challenge
A nightly factuality score drops six points, and a teammate wants to revert the model checkpoint. The evaluation uses a model judge with a string-match fallback, while different teams own the checkpoint, harness, templates and judge.
Compare the evidence available from the two runs before blaming the model. Choose checks that can distinguish a change in model behaviour from a change in what was tested, how it ran or how the answers were graded.
What you'll practise
Comparable runs
Establish which inputs and evaluation components changed between the results.
Failure localisation
Use item-level evidence to separate model behaviour from measurement errors.
Explain which evidence you would inspect and what each result would change.