Context Rot

Context rot describes the progressive loss of relevance, accuracy, or coherence in the situational information an AI system relies on to generate responses. This includes conversation history, cached embeddings, retrieved documents, and other inputs that collectively define what the model "knows" about its current task or user. As these inputs age, drift, or get truncated, the system's outputs become increasingly stale, inconsistent, or misaligned with current facts and user intent.

The phenomenon emerges from several intersecting pressures. Most large language models operate within fixed token windows, meaning older context is evicted as conversations grow longer—a process called context truncation or sliding-window eviction. Simultaneously, external knowledge sources indexed for retrieval-augmented generation (RAG) systems become outdated as the world changes, while cached embeddings may no longer accurately represent the documents they encode. In long-lived conversational agents, these forces compound: the model loses track of earlier commitments, contradicts itself across turns, and fails to maintain personalization. The result is elevated hallucination rates, degraded grounding, and eroded user trust.

Context rot is closely related to several well-studied ML phenomena. Catastrophic forgetting in continual learning describes how models lose previously acquired knowledge when trained on new data. Concept drift refers to the statistical shift between training distributions and live production data over time. Context rot can be understood as a runtime manifestation of both: the model's effective knowledge base decays not through weight updates but through the degradation of its dynamic inputs. Retrieval systems are particularly vulnerable, as document freshness and embedding fidelity both erode without active maintenance.

Mitigating context rot requires a combination of system design and operational discipline. Time-aware retrieval with explicit freshness signals, periodic reindexing of vector stores, hierarchical compression of long conversation histories, and memory-refresh mechanisms all help preserve context quality. Evaluation requires longitudinal testing—measuring embedding drift, retrieval relevance decay, and multi-turn consistency over extended sessions. As LLM deployments scale and session lifetimes grow, managing context rot has become a central concern in production AI engineering.