OpenAIVerified interview challenge
Investigate an annotator-quality warning
Determine whether a high disagreement rate is a reason to remove an annotator's labels from training.
Unlock with Full AccessThe challenge
An annotation dashboard labels one contributor the worst in the batch, and the team wants to discard their work before tomorrow's retrain. The available evidence includes assignment difficulty, disagreement with other annotators and independently adjudicated checks.
Investigate whether the ranking supports that decision. For each check, explain how the result would affect keeping, reweighting, relabeling or removing the annotations, including the consequences for the model trained on them.
What you'll practise
Comparable quality evidence
Account for the tasks, guidance and adjudicated samples behind the scores.
Data decisions
Explain which findings would justify changing how the labels are used.
Explain which evidence you would inspect and what each result would change.