Anthropic
NVIDIA
OpenAIBased on real interviews
Inference incident: a p99 latency cliff
Investigate a sudden rise in tail latency while median latency, throughput and error rates remain steady.
Unlock with Full AccessThe challenge
You are on call for a chat-completion service after a persistent latency step. A model update and a partial quantization rollout happened near the incident, but the timing alone does not establish the cause. Write a memo that guides the engineering investigation and gives leadership a defensible mitigation decision.
- Distinguish aggregate and customer impact
- The brief supplies endpoint-level measurements and a customer commitment. Identify the additional measurements needed before treating them as equivalent.
What you'll practise
Explaining the metric pattern
Propose mechanisms that could affect the latency tail without shifting every request.
Separating hypotheses
Choose measurements and tests that distinguish the recent changes and serving conditions.
Mitigation decisions
Balance recovery speed against quality, capacity and the risks of rollback.
Explain the evidence, next measurements and recommended response.