MA (Memory Attention)

MA (Memory Attention) is an attention variant that builds a token's value vector from a stored, per-layer memory lookup instead of a learned value projection over the current context. Jiale Kang introduced it in "Memory Attention" (arXiv:2609.28399, posted 23 September 2026, single-author, not yet peer-reviewed). The motivating question: standard attention recomputes values from contextual hidden states even when much of that content is tied to a token's identity rather than its context, and is therefore reusable everywhere the token appears.

MA forms each value as V = K + M. K is the ordinary contextual key, carrying context and prior-layer computation. M is a layer-specific memory vector retrieved by token ID rather than computed from the input. At inference, normalization can be folded into the memory tables, so building a value reduces to a lookup and an addition rather than a matrix projection. Because the memory is indexed by token ID, it can be offloaded to CPU with prefetching (MA-Offload). The author's prototype uses about 2.08 times a baseline model's total parameters, cuts GPU parameter storage by 7.38%, and keeps latency close to baseline.

Under matched training token budgets, MA improves language modeling loss and average downstream scores across the tested attention configurations, at the cost of the added memory parameters. The paper also sketches MA-Recall, which would drop the persistent value cache and reconstruct values on demand. The author reports this variant is still untested. Kang credits Engram as an inspiration and notes MA's relation to Muyu He's context-free value work and BlinkDL's DeepEmbedAttention from RWKV-8.