GoogleVerified interview challenge
Investigate confident but incorrect answers
Decide what to investigate when complaints rise but the nightly factuality benchmark stays flat.
Unlock with Full AccessThe challenge
A workplace-policy assistant is receiving more complaints about confident but incorrect answers, although its nightly factuality benchmark is flat. A recent release changed the model, retrieval ranking and citation display at the same time.
Before recommending a rollback, decide which customer evidence and release comparisons would separate these possible causes. Explain how each finding changes the response, including whether the issue is concentrated in a particular cohort.
What you'll practise
Conflicting signals
Reconcile support reports and benchmark results with coverage and traffic changes.
Rollback judgement
Link each finding to the component or cohort that needs action.
Explain which evidence you would inspect and what each result would change.