Fast weights

Fast weights are a class of adaptive parameters in neural networks that update on a much shorter timescale than conventional weights. This allows a network to encode transient information within a single sequence or task. Standard "slow" weights are adjusted gradually through backpropagation across many training examples, whereas fast weights change rapidly in response to recent inputs. They act as a short-term memory that complements the long-term knowledge stored in the network's primary parameters. Learning across these two timescales lets the network maintain stable general knowledge while adapting to immediate context.

The mechanism typically computes an outer product of recent hidden states or activity patterns and accumulates the result in a fast weight matrix. The matrix then modulates the network's activations. When a new input arrives, the fast weight matrix influences the network's response based on recent inputs without permanently altering the slow weights. This approach is related to associative memory and Hebbian learning, in which co-active neurons strengthen their connections transiently. In practice, fast weights decay over time or across steps, so they capture short-term dependencies rather than accumulating indefinitely.

Fast weights are especially relevant to recurrent neural networks and attention-based architectures, which must model short-range dependencies within a sequence. They can replace or complement mechanisms such as LSTMs and self-attention and provide a more biologically plausible account of working memory. The concept also connects to meta-learning, in which inner-loop adaptation across a task can be interpreted as a fast weight update. It is therefore foundational to approaches such as MAML and hypernetwork-based methods.

Geoffrey Hinton and colleagues originally proposed the idea in the late 1980s. It received renewed attention in 2016, when Jimmy Ba, Geoffrey Hinton, and collaborators demonstrated its utility in modern deep learning. Since then, fast weights have informed the design of memory-augmented networks, neural Turing machines, and efficient transformer variants. They connect classical associative memory with contemporary sequence modeling.