CLIP (Contrastive Language–Image Pre-training)

CLIP, short for Contrastive Language–Image Pre-training, is a multimodal neural network developed by OpenAI that learns to associate images with natural language descriptions. Rather than training on manually labeled datasets with fixed category sets, CLIP ingests hundreds of millions of image-text pairs collected from the internet, learning a shared embedding space where semantically related images and captions are pulled close together while unrelated pairs are pushed apart. This contrastive training objective allows the model to develop rich, flexible visual representations grounded in human language.

At inference time, CLIP performs classification by encoding both an image and a set of candidate text prompts—such as "a photo of a dog" or "a photo of a car"—and selecting whichever text embedding is most similar to the image embedding. This mechanism enables powerful zero-shot transfer: the model can classify images into entirely new categories simply by describing them in natural language, without any additional fine-tuning. The approach sidesteps the brittleness of traditional classifiers, which are locked to a fixed label vocabulary established at training time.

CLIP's significance extends well beyond image classification. Its image encoder has become a widely adopted backbone for downstream tasks including object detection, image generation guidance, and visual question answering. Models like DALL·E 2 and Stable Diffusion rely on CLIP embeddings to steer image synthesis toward text-specified content. The model also exposed important limitations, such as sensitivity to prompt phrasing and difficulty with fine-grained spatial reasoning, spurring a wave of follow-up research into improved vision-language pretraining strategies.

More broadly, CLIP exemplifies a paradigm shift in computer vision: moving away from curated, task-specific datasets toward large-scale self-supervised learning from naturally occurring web data. By treating language as a supervisory signal rather than a bottleneck, CLIP demonstrated that vision models could achieve remarkable generality and flexibility, influencing the design of nearly every major vision-language model that followed.