Meta
Google DeepMind
Anthropic
OpenAIBased on real interviews
Review a post-training experiment claim
Decide what a score improvement actually establishes when several parts of a training recipe changed.
Unlock with Full AccessThe challenge
A research memo argues that a new training recipe beats the deployed baseline and credits an auxiliary consistency loss for the improvement. The recipe also changes the data mix and training duration.
Review the comparison and the way the result was obtained. For each weakness, identify the conclusion it leaves unsupported and explain the experiment that would resolve it, without redesigning the proposed recipe.
What you'll practise
Causal attribution
Distinguish an improved result from evidence about which change caused it.
Fair comparison
Assess baseline treatment, repeated benchmark use and the controls needed for the claim.
Explain the issue, its consequence and the change you would make.