Attention

Attention is a neural network mechanism that lets models assign different weights to different parts of their input when producing an output, instead of treating every input element the same. Rather than compressing an entire input sequence into a single fixed-length representation, attention computes a set of scores that show how relevant each input element is to a given output step. Those scores get normalized into a probability distribution and used to form a weighted sum of input representations, which directs the model's focus toward the most contextually important information at each step.

The mechanics of attention usually involve three components: queries, keys, and values. A query represents what the model is currently trying to compute. Keys represent the available input elements. Values carry the actual content to be aggregated. Compatibility between a query and each key is measured, often via dot product, and the resulting scores are passed through a softmax function to produce attention weights. The final output is a weighted combination of the values, with greater weight on those most relevant to the query. This formulation, known as scaled dot-product attention, underpins the Transformer architecture introduced in 2017.

Attention became central to machine learning with Bahdanau et al.'s 2014 work on neural machine translation, which addressed the bottleneck problem in encoder-decoder models by allowing the decoder to look back at all encoder states rather than relying on a single compressed vector. Translation quality improved substantially on long sentences. The 2017 Transformer paper then showed that attention alone, without recurrence or convolution, could achieve state-of-the-art results, which made attention the dominant paradigm in sequence modeling.

Attention is the foundational operation in large language models such as GPT and BERT, and has been extended to computer vision, multimodal learning, and reinforcement learning. Multi-head attention runs several attention operations in parallel and concatenates their outputs, which lets models capture different types of relationships within data at the same time. Attention mechanisms have enabled models to scale to larger sizes while maintaining the ability to capture long-range dependencies that earlier architectures had trouble representing.