OpenAI
AnthropicBased on real interviews
Review a document-assistant launch evaluation
Find what an impressive internal accuracy result leaves unanswered about a document-assistant launch.
Unlock with Full AccessThe challenge
AskDocs answers questions about uploaded files, from contracts and invoices to scanned PDFs and tables. Leadership wants a broad launch, and the team has a promising accuracy result from its initial validation.
Review whether the evidence supports that launch across the customers and documents the product will actually encounter. Explain what each gap could let through to users and how you would improve the go-or-no-go decision.
What you'll practise
Evaluation coverage
Assess whether the test population represents the situations users will encounter.
Launch evidence
Identify the missing checks that could change a go/no-go recommendation.
Explain the issue, its consequence and the change you would make.