NVIDIA
AnthropicBased on real interviews
Investigate slow model inference
Identify what makes a long-context customer demo slow before committing to more GPU capacity.
Unlock with Full AccessThe challenge
A customer's long-context demo is tomorrow, and a teammate wants to add GPUs because inference feels too slow. The service already uses continuous batching, KV caching, prefix caching and routing that keeps a tenant on the same nodes.
No one has pulled the latency breakdown yet. Choose the measurements that would distinguish a capacity problem from delays elsewhere in this serving setup, and explain how the results would change your recommendation.
What you'll practise
Performance measurement
Separate the phases and workload characteristics behind the latency complaint.
Targeted interventions
Connect evidence to the choice of a fix instead of assuming more hardware is needed.
Explain which evidence you would inspect and what each result would change.