AnthropicBased on real interviews
Assess a historical-chat fine-tuning dataset
Find out whether a large chat archive is the right starting point for a better support model.
Unlock with Full AccessThe challenge
A support team wants to fine-tune on two million historical chats. The archive spans several products, languages and regions, mixes human and older-bot conversations, and includes policies that have changed since the chats took place.
The current system struggles with recent policy changes and unusual cases. Decide what needs to be established about the archive, its outcomes and its permitted use before anyone treats its size as a reason to start training.
What you'll practise
Desired behaviour
Determine which conversations represent the responses the team now wants.
Evaluation readiness
Identify whether the archive and outcome evidence can support a trustworthy comparison.
Explain how the answers would change the decision or scope.