OpenAI
NVIDIABased on real interviews
Diagnose instability after scaling training
Find what changed when a stable training run moved from one node to a larger cluster.
Unlock with Full AccessThe challenge
A fine-tuning run is stable on eight GPUs but develops recurring loss spikes when scaled to 64. The model and dataset are unchanged, and aggregate throughput is close to the expected scaling result.
A teammate wants to lower the learning rate and restart. Choose the checks that would establish what changed when the run crossed nodes, using data shards, training settings and per-rank evidence before deciding whether retuning is the right response.
What you'll practise
Scale-dependent changes
Identify which effective training conditions change when the GPU count increases.
Per-rank evidence
Look beyond aggregate throughput and rank-zero loss when isolating the problem.
Explain which evidence you would inspect and what each result would change.