InfoNCE

InfoNCE (Information Noise-Contrastive Estimation) is a contrastive loss function introduced by van den Oord, Li, and Vinyals (DeepMind, 2018). It lower-bounds the mutual information between two related views of data, typically (context, future), (anchor, positive), or (sample, augmented sample), by reformulating the mutual information bound as a categorical cross-entropy over a positive sample against a set of negative samples drawn from a proposal distribution.

The loss takes the form -E[log(f_k / sum_j f_j)], where f_k is the unnormalized similarity score between the positive pair and f_j ranges over the negative samples. InfoNCE is bounded above by log(N), where N is the number of negative samples. Increasing the number of negatives tightens the bound on the true mutual information.

InfoNCE is the loss used in nearly all modern self-supervised and contrastive learning methods. It was first applied in Contrastive Predictive Coding (CPC) for audio and image representations, then adopted by SimCLR, MoCo, CLIP, and the family of contrastive vision-language models. The loss is closely related to MINE, since both estimate mutual information through gradient-based optimization. InfoNCE uses a multi-sample classification objective (N-way softmax) rather than the Donsker-Varadhan bound, which makes it simpler to implement, less biased, and easier to scale to large embedding spaces.

The canonical paper is "Representation Learning with Contrastive Predictive Coding," arXiv:1807.03748 (July 2018).