Deep InfoMax (DIM)

Deep InfoMax (DIM) is a representation learning framework introduced by Hjelm, Fedorov, Lavoie-Marchildon, Grewal, Bachman, Trischler, Bengio (Mila, 2018) that maximizes mutual information between the high-level (global) feature vector of an encoder and its lower-level (local) convolutional features, in order to learn representations that retain information about the input across scales.

The original formulation uses MINE (Mutual Information Neural Estimation) as the differentiable mutual information estimator between local and global features. Two variants are introduced: (1) the basic version (DIM) maximizes MI between global features and all local features simultaneously, and (2) the contrastive version (DIM-Loc) trains a discriminator to distinguish local features from a global feature against local features from other images, providing stronger gradient signal. The encoder, discriminator, and MI estimator are trained jointly end-to-end.

DIM demonstrated that maximizing mutual information between scales produces useful representations for downstream tasks on MNIST, CIFAR, and ImageNet, and outperformed contemporary self-supervised baselines including variational autoencoders and bi-prediction GANs. It established mutual information maximization as a viable objective for unsupervised representation learning and inspired a wave of follow-up work — including Contrastive Predictive Coding's InfoNCE bound, Deep Graph Infomax (DGI) for graph-structured data, Spatio-Temporal Deep Graph Infomax, and Video Deep InfoMax. Subsequent work has critiqued whether the gains truly come from MI maximization versus encoder inductive biases.

The canonical paper is Learning deep representations by mutual information estimation and maximization, arXiv:1808.06670 (August 2018, later ICLR 2019).