KV Cache

A KV cache (key-value cache) is a memory buffer used during autoregressive generation with transformer language models. In self-attention, each token's key and value vectors depend only on that token and the model's fixed weights, not on later tokens, so once computed they never change. Without caching, generating each new token would require recomputing the keys and values for every earlier token in the sequence, work that grows with the length of everything generated so far. A KV cache stores these vectors as they are produced and reuses them on every later step, so generating a new token only requires computing that token's own query, key, and value and attending over the cached keys and values of everything before it.

This turns the per-token cost of generation from growing with the full sequence length into work that grows only with the newly added token. The tradeoff is memory that scales with sequence length, number of layers, and model size, since each transformer layer keeps its own cache. The technique became a standard part of transformer inference as autoregressive language models scaled up in the years following the original Transformer architecture (Vaswani et al., 2017), and it is documented as a core serving optimization in major inference libraries such as Hugging Face Transformers. The cache grows with every generated token and can dominate GPU memory at long context lengths. Later work on cache compression, quantization, CPU offloading, and fixed-size ("static") caches addresses this memory cost.

The KV cache holds a transformer's running, layer-by-layer representation of everything it has read or generated, and researchers have started treating it as more than an efficiency trick. Some 2025-2026 work manipulates or transfers KV-cache contents directly between different models or components, using it as a channel for passing internal state rather than only a cache, as in Cache-to-Cache communication between large language models.