AnthropicVerified interview challenge
Build a longest-match tokenizer
Turn text into token IDs by finding the longest available vocabulary match at each position.
Unlock with Full AccessThe challenge
Build a tokenizer that reads text from left to right, resolves overlapping vocabulary entries and handles characters outside its vocabulary. The full problem defines how unknown text should be represented and gives the constraints your implementation must handle.
What you'll practise
Greedy matching
Choose the correct token when several vocabulary entries fit.
Unknown text
Keep output consistent when the vocabulary cannot cover the input.
Search efficiency
Handle longer inputs and larger vocabularies without unnecessary work.
solution.pyPython 3.13
def longest_match_tokenize(
text, vocabulary
):
# Your implementation goes here
passTest cases
Run your code to see results.
Full instructions, starter code and test cases included with Full Access.