Long-Horizon Memorization

Long-horizon memorization is an evaluation setting for continual learning, introduced by Zheyuan Zhang, Alvin Zhang, Daniel Khashabi, and Tianmin Shu of Johns Hopkins University in "Continual Learning Mechanisms Compose for Long-Horizon Memorization" (arXiv:2609.06986, September 2026). A language model learns 100 query-answer tasks in sequence through continual supervised fine-tuning, without replaying raw examples from earlier tasks and without receiving a task identifier at inference. The goal is to measure how much of that earlier information survives in the model's parameters after many further updates, not how well the model generalizes to new tasks.

The paper reports that naive sequential fine-tuning collapses to 1.2% average final retention across 100 tasks. No single continual-learning mechanism it tests, whether replay, self-distillation, or importance-based weight regularization, prevents this collapse on its own. Combining three complementary "anchors" (data, function, and weight) with a low-rank adaptation rule called merged LoRA raises average final retention to 34.9%, a 28-fold improvement. It also extends the "memory half-life," the number of further tasks before a fact's recall accuracy drops by half, from a handful of tasks to several dozen.

The setting differs from more common continual-learning or continual-pretraining benchmarks, which typically measure transfer or task performance across a stream of heterogeneous tasks or corpora rather than exact retention of a large, fixed set of facts. It also differs from sequential model-editing benchmarks, which study targeted individual edits rather than ordinary supervised fine-tuning across many tasks. Long-horizon memorization is a stress test for how well parameter updates in a language model can serve as durable memory.