nGPT (Normalized Transformer)

nGPT, or the Normalized Transformer, is a transformer architecture variant in which all hidden state representations are constrained to lie on the surface of a unit hypersphere. Rather than applying normalization only at specific points within the network (as in standard layer normalization), nGPT enforces this geometric constraint throughout the model, treating the learning process as movement along a curved manifold. This structural choice fundamentally changes how information flows through the network and how gradients propagate during training.

The core mechanism works by normalizing weight matrices and hidden states so that every representation maintains unit norm. Updates are performed using a form of geodesic interpolation on the hypersphere rather than standard additive gradient steps. This means the model learns by rotating and blending directions in representation space rather than shifting magnitudes, which has the practical effect of making the optimization landscape more uniform and predictable. The architecture also introduces learnable scaling factors that allow the model to modulate the influence of each component without breaking the normalization constraint.

The practical benefits of this design are significant. Empirical results show that nGPT can reach the same validation loss as a standard transformer in substantially fewer training steps — sometimes four to ten times fewer — which translates directly into reduced compute costs. The hyperspherical constraint also appears to improve training stability, reducing sensitivity to learning rate choices and diminishing the risk of gradient explosion or collapse that can plague deep transformer stacks.

nGPT matters because it demonstrates that rethinking the geometric structure of representations, rather than simply scaling model size or data, can yield meaningful efficiency gains. As the cost of training large language models becomes a central concern in AI research and deployment, architectural innovations like nGPT offer a path toward more resource-efficient systems. The approach also opens theoretical questions about why hyperspherical geometry is beneficial, connecting practical deep learning to ideas from differential geometry and optimization on manifolds.