MetaBased on real interviews
Review a recommender for travellers in a new city
Check whether a recommender's offline results will hold up for travellers with no local history.
Unlock with Full AccessThe challenge
A travel app recommends places in a city a user has never visited. It knows their preferences from other cities and can use venue metadata, availability, distance and past recommendation impressions.
Review the proposed training, evaluation and rollout plan. The challenge is to connect apparent offline success to the information actually available when a traveler needs a recommendation, and to the feedback the ranking will generate after launch.
What you'll practise
Prediction-time evidence
Distinguish usable features from information that only arrives after the recommendation.
Evaluation validity
Check whether labels and data splits represent the traveller's actual choices.
Explain the issue, its consequence and the change you would make.