Anthropic
OpenAIBased on real interviews
Evaluate autonomous task execution
Design the evaluation that will inform whether a model can be deployed to take multi-step actions for users.
Unlock with Full AccessThe challenge
A newly post-trained model can browse, execute code, call APIs and operate on files. A deployment board needs an evaluation of its autonomous task execution before deciding where it can be used.
Design a runnable evaluation programme within a 90-day window. Keep evidence of useful task competence separate from evidence of dangerous capabilities, and explain how task sourcing, elicitation, scoring and uncertainty affect the deployment recommendation.
What you'll practise
Evaluation methodology
Choose tasks and measurements that answer the deployment question.
Confound control
Explain how your design handles misleading signals and incomplete coverage.
Decision thresholds
Connect evaluation results to an explicit recommendation and stated governance assumptions.
Write your reasoning and submit it for feedback.