OpenAIVerified interview challenge
Transformer Debugging
Find four bugs in a small Transformer and make its cached decoder agree with a full forward pass.
Unlock with Full AccessThe challenge
Repair the supplied PyTorch model and next-token loss, then reuse keys and values as new tokens arrive. Both parts share one Python file, with training and sampling helpers for debugging. Choose guided practice or a 60-minute timed mock.
What you'll practise
Model debugging
Trace attention, positions and the training objective through a small decoder.
Gradient checks
Check that a working forward pass still trains the intended task.
Cached decoding
Reuse per-layer state while preserving the model's predictions.
Current partFurther parts unlock as you progress
solution.pyPython 3.13
# Build on your implementation
# Run checks for the current part
# Full starter code includedCurrent-part checks
Run your code to see results.
The complete sequence, starter code and checks are included with Full Access.