OpenAI
Anthropic
MetaBased on real interviews
Design memo: production eval suite for structured-data extraction
Replace a small manual evaluation with a system that can catch regressions before customers do.
Unlock with Full AccessThe challenge
A document-extraction feature returns structured data in each customer's requested schema. Recent releases introduced date-format and nested-field failures that the existing spreadsheet checks missed. Design an evaluation suite that supports release decisions and ongoing monitoring within the supplied review, runtime and cost budgets.
What you'll practise
Evaluation coverage
Choose examples that represent the customers, documents and schemas the feature serves.
Metrics and release decisions
Define what the checks measure and how their results affect a proposed release.
Operating the evaluation
Plan how reviews and monitoring work within the team's available time and budget.
Explain how the team should test changes and monitor the feature.