Multi-Token Prediction

Multi-token prediction is a language model training and inference strategy in which a model is trained to predict several future tokens at once, rather than forecasting only the immediately next token. In the standard autoregressive paradigm, a model conditions on all previous tokens to predict the next single token, repeating this process sequentially. Multi-token prediction extends this by adding auxiliary prediction heads that forecast tokens two, three, or more steps ahead from the same hidden representation, effectively asking the model to plan further into the future at each position during training.

The mechanism typically involves attaching multiple independent output heads to a shared transformer backbone, where each head is responsible for predicting a token at a different future offset. During training, losses from all heads are combined, encouraging the model to develop internal representations that encode richer, more forward-looking contextual information. At inference time, the additional heads can optionally be used for speculative decoding — generating candidate future tokens in parallel and verifying them, which can substantially reduce the number of sequential forward passes required and improve throughput.

The significance of multi-token prediction lies in two distinct benefits. First, it acts as a richer self-supervised training signal: by forcing the model to simultaneously account for multiple future tokens, the representations it learns tend to capture higher-level semantic structure rather than low-level token-by-token statistics. Empirical results have shown improvements on code generation and reasoning benchmarks when models are trained with this objective. Second, the inference-time speedup from speculative decoding can be substantial on hardware where parallel computation is cheap, making deployment of large models more practical.

Interest in multi-token prediction as a deliberate training objective grew notably around 2024, when Meta AI published research demonstrating that training large language models with four-token prediction heads improved both sample efficiency and downstream task performance. This distinguished the approach from earlier speculative decoding work, which had focused purely on inference acceleration without modifying the training objective. The technique is now considered a promising direction for making next-generation language models both more capable and more efficient.