MINE (Mutual Information Neural Estimation)

MINE (Mutual Information Neural Estimation) is a method for estimating the mutual information I(X;Y) between two high-dimensional continuous random variables by training a neural network classifier through gradient descent on a Donsker-Varadhan representation of the KL divergence. The estimator is linearly scalable in both dimensionality and sample size, trainable end-to-end via backpropagation, and strongly consistent (converges to the true MI as sample size grows). It was introduced by Belghazi, Baratin, Rajeswar, Ozair, Bengio, Courville, and Hjelm at Mila in 2018.

The method reformulates mutual information as the supremum over a function family of an expectation: I(X;Y) = sup_T E_{p(x,y)}[T(x,y)] - log(E_{p(x)p(y)}[e^{T(x,y)}]). A neural network T plays the role of the critic, and gradient descent over its parameters provides a tight lower bound on the true MI. The lower bound is tight in expectation but biased for finite samples, motivating biased gradient estimators and the use of an exponential moving average baseline in practice.

MINE unlocked the practical use of mutual information as an optimization target in deep learning. It directly enabled Deep InfoMax (Hjelm 2018), which maximizes MI between local and global features for representation learning, and the Information Bottleneck principle in deep networks. Follow-up work includes conditional MI estimators (CMI), normalized MINE (NMINE) for variance reduction, and diffusion-based MI estimators (MINDE). The technique is foundational for self-supervised and contrastive representation learning, and is closely related to InfoNCE — both bound mutual information from below, but through different variational families.

The canonical paper is arXiv:1801.04062 (January 2018, ICLR 2018 workshop).