AmazonBased on real interviews
Monitor a production support assistant
Build monitoring that can tell the team whether its support assistant is ready for more traffic.
Unlock with Full AccessThe challenge
A support assistant drafts replies that agents can send, edit, discard or escalate. It is serving a small share of English traffic, and the team wants to expand exposure and add Spanish if the pilot performs well.
Verified resolutions arrive several days later, while immediate customer ratings are sparse. Build a compact monitoring framework from these outcomes and the recorded agent actions, model versions, retrieved documents, latency and usage. Explain what decision each measure supports.
What you'll practise
Outcome signals
Use delayed resolution evidence and agent actions without over-relying on sparse ratings.
Operational decisions
Define the slices, time windows and guardrails that would guide expansion.
Define each measure and the decision it would support.