OpenAI
Anthropic
NVIDIABased on real interviews
Spot the bugs: transformer inference
Find why a small Transformer implementation can run successfully and still produce incorrect attention or generations.
Unlock with Full AccessThe challenge
The code includes attention, prompt processing and a cached decoding step that generates one token at a time. Imports and pretrained weights are supplied, so the review centres on how those operations fit together. Trace the implementation and explain how each fault changes the model's output.
A tensor can have a plausible shape even when the calculation is wrong. This exercise asks you to connect the mathematics to the code and describe a focused correction. You submit a written diagnosis rather than a replacement implementation.
What you'll practise
Attention calculations
Follow how token representations become attention weights and an output.
Cached generation
Track what carries forward from the prompt into each decoding step.
Explaining a defect
Connect a local code mistake to the output it changes, with a correction that addresses that cause.
Describe the fault, its effect and the fix.