Transformer
A neural network architecture using self-attention to process sequential data in parallel.
The Transformer is a neural network architecture introduced in the 2017 paper "Attention Is All You Need." Rather than processing tokens one at a time like recurrent networks do, the Transformer uses self-attention to compute relationships between all positions in a sequence simultaneously. Each token attends to every other token, producing weighted representations that capture context regardless of distance. This matters for long sequences because RNNs struggle to propagate information across them. The original architecture pairs an encoder, which builds contextual representations of the input, with a decoder that generates output token by token while attending to both its own prior outputs and the encoder's representations.
The self-attention mechanism projects each token into three vectors: query, key, and value. It then computes dot-product similarities between queries and keys to determine how much each token should attend to every other. These attention scores are scaled, softmax-normalized, and used to produce a weighted sum of value vectors. Multiple attention heads run in parallel, each learning to focus on different types of relationships, and their outputs are concatenated and projected. Positional encodings are added to token embeddings to inject sequence order information, since the architecture itself is otherwise permutation-invariant. Stacked layers of multi-head attention and feed-forward sublayers, each wrapped with residual connections and layer normalization, build increasingly abstract representations.
The Transformer became the backbone of virtually every major language model, including BERT, GPT, T5, and their successors, enabling progress in translation, summarization, question answering, and code generation. Its parallelism makes it well suited to large-scale training on modern GPU and TPU hardware, supporting the scaling laws that underpin today's large language models. Beyond NLP, the architecture has been adapted for vision (Vision Transformers), audio, protein structure prediction, and reinforcement learning.
Sources
- Attention Is All You Need
arXiv · Jun 12, 2017
- Transformer
Wikipedia
Research this in Signals
Signals turns a topic into a sourced research record you can inspect and rerun. Your first scan is free, and this one starts with Transformer already loaded, so edit it or scan as is.
