NVIDIAVerified interview challenge
Recover a 500-node training run
Keep a large training run on schedule when a failed node stalls every other rank.
Unlock with Full AccessThe challenge
A synchronous training run spans hundreds of nodes, and one failed rank can block the entire job. Checkpoint pauses, lost work and replacement delays are consuming time the run cannot easily recover.
Propose a failure-detection, checkpoint and recovery design using the supplied operating measurements. A faster restart is useful only if the restored model, optimizer and data position still represent a consistent training step. Explain that correctness boundary alongside the expected wall-clock cost.
What you'll practise
Checkpoint correctness
Explain how rank shards and training state become one usable snapshot.
Recovery timing
Account for detection, lost work, allocation and restore costs.
Failure testing
Plan fault injection that tests both recovery speed and resumed-run correctness.
Write your reasoning and submit it for feedback.