AnthropicBased on real interviews
Review a GPU batching plan
Find where an inference batching proposal could miss deadlines, waste capacity or lose requests.
Unlock with Full AccessThe challenge
An interactive text-generation API must start responding within 750 milliseconds, while traffic and generation lengths vary widely. The team has proposed a batching plan and wants it reviewed before implementation.
Read the scheduler and request-lifecycle decisions together. A plan that keeps the GPU busy can still leave callers waiting, waste work after disconnects or mishandle a failed request. Explain the consequences of the weaknesses you find and propose focused changes.
What you'll practise
Queueing and deadlines
Assess how batching decisions affect waiting callers.
Request reliability
Trace how overload and failures interact with completion and capacity use.
Explain the issue, its consequence and the change you would make.