AnthropicVerified interview challenge
Build and scale a concurrent web crawler
Take a crawler from sequential graph traversal to concurrent execution and a distributed design discussion.
Unlock with Full AccessThe challenge
Build a crawler over a supplied page graph, initially visiting pages one at a time on the seed host. Then keep the same behaviour while making traversal concurrent.
The final part asks how your implementation would operate across machines. It gives you a concrete design to discuss, including discovery, duplicate work and knowing when the crawl is complete. Page access is supplied by the exercise, with no live network crawling.
What you'll practise
Shared work coordination
Keep discovery, deduplication and completion correct when workers run together.
Distributed follow-up
Explain which assumptions change when work moves beyond one process.
Current partFurther parts unlock as you progress
solution.pyPython 3.13
# Build on your implementation
# Run checks for the current part
# Full starter code includedCurrent-part checks
Run your code to see results.
The complete sequence, starter code and checks are included with Full Access.