L2M (Large Memory Model) is a decoder-only Transformer architecture augmented with an explicit, often differentiable auxiliary memory module that provides persistent, content-addressable storage for intermediate representations, retrieved states, or relational facts so the model can perform multi-step reasoning, maintain dialogue/world state, and synthesize information distributed across very long contexts that exceed the model's local attention span. In practice the memory is interfaced via read/write primitives (content-based or sparse addressing, learned keys, gating) and trained either end-to-end with auxiliary losses (e.g., retrieval supervision, reconstruction, or contrastive objectives) or combined with retrieval-augmentation pipelines; this design reduces reliance on exponentially large context windows by offloading long-lived information to the memory, enables chaining of reasoning steps by storing intermediate latent states, and supports relational argumentation by linking entities across disparate positions. Architecturally L2M variants borrow ideas from memory-augmented neural networks (Neural Turing Machines, Memory Networks), compressive memories, and retrieval-augmented generation, while addressing practical concerns—memory capacity, read/write latency, memory consolidation/compression, and consistency—via sparse access, hierarchical memory tiers, and learned compression or eviction strategies. Applications where L2M shows value include long-document question answering, multi-turn dialogue with persistent state, complex program synthesis or verification over large codebases, and multi-hop scientific or legal reasoning where evidence is distributed across long inputs.
First usages and concrete proposals for memory-augmented, Transformer-based decoders appeared in the early-to-mid 2020s; the design gained broader popularity and experimental traction around 2023–2024 as long-context modeling, retrieval-augmented generation, and compressive-memory approaches demonstrated practical benefits for reasoning and long-document tasks.
Key contributors include foundational work on attention and Transformers (Vaswani et al.), early memory-augmented network research (Graves on Neural Turing Machines; Weston et al. on Memory Networks), and later advances in long-range and memory-aware Transformers such as Rae et al. (Compressive Transformer), Beltagy et al. (Longformer), Zaheer et al. (BigBird), and retrieval/RAG-style systems (Lewis et al.); major research groups at DeepMind, OpenAI, Meta, and leading academic labs have driven much of the architecture engineering, benchmarking, and deployment work that shaped L2M designs.