KL (Kullback–Leibler) Divergence

KL divergence is an asymmetric measure from information theory that quantifies how much one probability distribution P differs from a reference distribution Q. Formally, it is defined as the expected value of the log ratio of P to Q under P, and it can be interpreted as the amount of information lost when Q is used to approximate P. It is always non-negative and equals zero only when the two distributions are identical, making it a principled way to measure distributional discrepancy.

In machine learning, KL divergence appears throughout probabilistic modeling and optimization. In variational inference, it serves as the penalty term that keeps an approximate posterior close to a chosen prior, forming the basis of the evidence lower bound (ELBO) used to train variational autoencoders. In supervised learning, minimizing KL divergence between a model's predicted distribution and the true data distribution is mathematically equivalent to maximizing log-likelihood, connecting it directly to standard training objectives. It also underpins knowledge distillation, where a student model is trained to match the soft output distribution of a teacher model.

A critical property of KL divergence is its asymmetry: KL(P‖Q) is not generally equal to KL(Q‖P). This means the choice of which distribution is treated as the reference matters significantly. When P is the true distribution and Q is the approximation, minimizing KL(P‖Q) encourages Q to cover all regions where P has mass (mean-seeking behavior), while minimizing KL(Q‖P) encourages Q to concentrate on the most probable modes of P (mode-seeking behavior). Understanding this asymmetry is essential for choosing the right objective in generative modeling and approximate inference.

Beyond these core applications, KL divergence connects to a broader family of divergence measures, including the Jensen–Shannon divergence (a symmetrized version) and the f-divergences used in generative adversarial network theory. Its deep ties to entropy, mutual information, and cross-entropy make it one of the most foundational quantities in both information theory and modern machine learning.