Anthropic
OpenAIBased on real interviews
Spot the bugs: ML eval harness
Review an LLM evaluation pipeline whose accuracy score looks precise but cannot be trusted.
Unlock with Full AccessThe challenge
The harness retrieves documents, generates answers and asks a language model to judge those answers against a reference set. Its final output is a single accuracy number. Inspect the code behind that number and explain which implementation choices prevent it from being reliable evidence.
The task follows the whole measurement path, from the examples and context supplied to the model through to the judgment counted as a success. You need to explain the effect of each fault and a targeted fix, without replacing the pipeline.
What you'll practise
Evaluation setup
Follow how examples, retrieved context and model calls contribute to the result.
Automated judging
Examine the step that turns a generated answer into a recorded success or failure.
Interpreting accuracy
Explain what a reported score can establish once its measurement process is understood.
Explain why the result is unreliable and how to fix it.