Next-Latent Prediction (NextLat)

Next-Latent Prediction, abbreviated NextLat, is a self-supervised learning objective that trains a transformer to predict its own next latent state — the representation the model would produce at the next step — rather than the next surface token. The motivation is that next-token prediction is myopic: a model optimizing for the next visible symbol ends up encoding mostly local continuation patterns, with reasoning depth emerging only incidentally through scale. By shifting the prediction target from the token to the model's own internal representation, NextLat forces the network to learn a more compact, more abstract internal dynamics — a world model in the sense of a simulator the model can roll forward in its own latent space.

Mechanically, NextLat adds a lightweight predictor head to the transformer that, given the current hidden state, estimates the hidden state the model would produce on the next step. The loss is computed in representation space — typically a normalized dot product or a small regression head on the projected next state — and is added to the standard next-token cross-entropy loss. The result is a multi-task objective: the model must still produce the right next token, but it is also rewarded for building a latent space whose own evolution is predictable. Training proceeds exactly as for any transformer; at inference time, the predictor head is discarded and the model behaves identically to a standard causal language model, with the reasoning benefits carried implicitly in the structure of the learned representations.

The advantage of NextLat over pure next-token prediction is the inductive bias toward deeper reasoning: because the model is rewarded for representing a state from which the next state can be predicted, it has an incentive to compress world knowledge into a form that supports multi-step inference rather than only local fluency. Empirically the approach has been shown to improve performance on reasoning benchmarks and to produce internal representations that are more linear and more separable by abstract features. The cost is modest — a small additional head and a second loss — but the gains depend on careful weighting of the two objectives, and the prediction target (the next hidden state) is not directly supervised, so the technique shares the under-specification issues of other self-distillation objectives. The relationship to JEPA is structural: JEPA predicts masked regions in representation space for representation learning, while NextLat predicts the next representation in a sequence for training a generative reasoner.

Open questions include how to scale the auxiliary loss and the predictor head to very large models without destabilizing training, whether the latent-prediction signal is still useful when combined with explicit chain-of-thought supervision, and whether the internal representations NextLat produces have properties that transfer to downstream agents and world models. There is also active debate about whether predicting the next latent state is the right auxiliary objective at all, or whether predicting longer-range future states — or even counterfactual states — would yield a stronger inductive bias. The technique is recent and the literature on its scaling properties, robustness, and downstream uses is still emerging.