OpenAIVerified interview challenge
Clean labels from noisy annotators
Implement the specified rules for identifying unreliable annotators and rebuilding cleaned sample labels.
Unlock with Full AccessThe challenge
Each sample contains votes from several annotators. Measure each person's agreement against their peers, apply the provided quality thresholds and produce cleaned labels from the remaining votes. Follow the contract for ambiguous comparisons and deterministic output. This is an implementation exercise for a defined policy, rather than a claim that majority agreement proves label quality.
What you'll practise
Peer comparisons
Keep an annotator's own vote separate from the consensus used to assess it.
Threshold and tie handling
Apply the policy consistently when evidence is sparse, tied or exactly at a boundary.
solution.pyPython 3.13
# Your implementation goes here
# Run examples and inspect results
# Full starter code includedTest cases
Run your code to see results.
Full instructions, starter code and test cases included with Full Access.