AnthropicVerified interview challenge
Design synchronous GPU request batching
Increase inference throughput without leaving synchronous callers waiting beyond their deadlines.
Unlock with Full AccessThe challenge
An interactive API currently sends requests to GPUs individually. Its public beta needs more throughput from the existing fleet while still returning responses within tight deadlines, including during bursts or a GPU outage.
Use measured batch profiles to design the request lifecycle, queueing and dispatch policy. Carry that design through timeouts, retries and degraded capacity, then propose how to introduce it before the beta without changing the model itself.
What you'll practise
Capacity and latency
Use throughput and execution-time measurements for the different planning questions they answer.
Request lifecycle
Explain how deadlines, retries and queued work interact.
Degraded operation
Define admission and recovery behaviour when serving capacity drops.
Write your reasoning and submit it for feedback.