Review a misuse detector before launch

The challenge

A safeguards team wants to replace hand-written rules with a prompt classifier on an API handling millions of requests a day. Clear policy violations are rare, and the human review team can inspect only a small fraction of traffic.

Review the proposed evaluation and rollout plan. Assess what its headline result says about missed violations and blocked legitimate users, and whether the evidence remains trustworthy once the detector starts influencing its own future training data.