NVIDIABased on real interviews
Investigate lost training throughput
Locate the lost training time when a busy GPU dashboard hides a large drop in useful throughput.
Unlock with Full AccessThe challenge
An input-pipeline refactor reduces useful training throughput, even though the loss remains stable and the dashboard shows high GPU-active time. Step times have worsened, especially in the tail.
You have per-rank timelines covering data queues, copies, kernels and collectives, along with padding statistics and the previous build. Plan the comparisons that would locate the lost time before committing to another large run.
What you'll practise
Useful work measurement
Distinguish device activity from progress on non-padding training tokens.
Timeline diagnosis
Compare data, copies, kernels and collectives to identify the relevant bottleneck.
Explain which evidence you would inspect and what each result would change.