xAIVerified interview challenge
Build a dynamic batching engine
Keep a generation batch useful as requests arrive with different lengths and finish at different times.
Unlock with Full AccessThe challenge
Implement an inference engine around a supplied model that returns one token per batch row. Requests have different prompt lengths and generation limits, with callbacks that must receive the correct result.
Once the common engine works, extend it for fixed-size padded calls or per-request stop sequences. The variants exercise different assumptions about when a request finishes and which rows still need model work. Guided practice lets you try both separately.
What you'll practise
Request lifecycle
Keep each request's tokens, completion and callback associated with the right state.
Batch evolution
Adapt scheduling to padding or stopping rules without mixing requests.
# Build on your implementation
# Run checks for the current part
# Full starter code includedRun your code to see results.