Autoregressive

An autoregressive (AR) model is a statistical framework that predicts the next value in a sequence as a linear combination of its previous values. Formally, an AR model of order p expresses the current observation as a weighted sum of the p preceding observations plus a noise term, where the weights — called autoregressive coefficients — are estimated from training data. The order p determines how far back in time the model looks, and selecting it appropriately is critical to balancing model complexity against predictive accuracy.

In machine learning, autoregressive thinking has expanded well beyond classical time series into a broad modeling paradigm. Modern autoregressive models — including PixelCNN for images, WaveNet for audio, and large language models like GPT — treat sequence generation as a chain of conditional probability estimates: each output token or value is sampled from a distribution conditioned on everything generated so far. This decomposition follows directly from the probability chain rule and allows arbitrarily complex distributions to be modeled without assuming independence between elements.

Training autoregressive models is relatively straightforward because ground-truth context is available at every step during training — a technique called teacher forcing. At inference time, however, the model must consume its own previous outputs, which can cause errors to compound over long sequences. This exposure bias is an active area of research, with approaches like scheduled sampling and reinforcement learning fine-tuning proposed as mitigations.

Autoregressive models matter because they offer a principled, tractable way to assign exact likelihoods to sequences, unlike generative adversarial networks or certain latent variable models. This property makes them valuable for density estimation, anomaly detection, and controlled generation. Their dominance in natural language processing — where the autoregressive transformer architecture underpins most state-of-the-art systems — has made understanding this concept essential for anyone working in modern AI.