---
title: Continual Learning
type: vocabulary
url: "https://www.envisioning.com/vocab/continual-learning"
summary: A machine-learning paradigm in which a model learns a sequence of tasks over time, accumulating knowledge across them without forgetting what it learned earlier.
year: 2019
generality: 0.55
---

# Continual Learning

A machine-learning paradigm in which a model learns a sequence of tasks over time, accumulating knowledge across them without forgetting what it learned earlier.
Continual learning (also called lifelong learning or incremental learning, depending on the community) is the machine-learning paradigm in which a model is exposed to a stream of tasks or data distributions over its lifetime, rather than being trained once on a fixed dataset. The defining challenge is catastrophic forgetting: standard gradient-based training on a new task tends to overwrite the parameters that supported prior tasks, causing accuracy on earlier tasks to collapse. Continual-learning research designs training procedures, architectures, and replay mechanisms that mitigate this trade-off so that performance on the full task sequence improves monotonically rather than degrading over time.

Approaches are typically grouped into three families: (i) regularization-based methods that penalize changes to parameters deemed important for prior tasks (for example, Elastic Weight Consolidation, Synaptic Intelligence); (ii) replay- or rehearsal-based methods that maintain a buffer of stored or generated examples from past tasks and interleave them with new training; and (iii) architectural or parametric-isolation methods that allocate dedicated sub-networks or capacity to each task. The term "continual learning" consolidated in its modern machine-learning sense through Parisi et al.'s 2019 review "Continual Lifelong Learning with Neural Networks," which unified usage across the prior lifelong-learning, incremental-learning, and never-ending-learning literatures. The paradigm has gained renewed urgency with large language models: training-time continual learning (continued pre-training on new corpora) and inference-time continual learning (in-context adaptation across a session) are now active research directions and the basis of several recent industry bets, including Sutskever-era Safe Superintelligence's reported first model.

---
Source: Envisioning — Technology Research Institute (https://www.envisioning.com/vocab/continual-learning)