Embedding
A dense vector representation that encodes semantic relationships between discrete items.
An embedding is a learned mapping that transforms discrete, high-dimensional objects—such as words, users, or products—into dense, continuous vectors in a lower-dimensional space. The core principle is that geometric relationships in this vector space reflect meaningful semantic or relational structure in the original data. Items that are conceptually similar end up close together, while dissimilar items are farther apart. This makes embeddings a powerful bridge between symbolic, categorical data and the numerical computations that machine learning models require.
Embeddings are typically learned as part of a neural network's training process. A lookup table, often called an embedding matrix, assigns each discrete token or entity a trainable vector. As the model optimizes for its primary objective—predicting the next word, classifying sentiment, or ranking recommendations—the embedding vectors are updated via backpropagation to capture the statistical regularities in the training data. The result is a compact representation that encodes rich contextual information without being explicitly programmed. Word2Vec, introduced in 2013, demonstrated this dramatically by showing that vector arithmetic on word embeddings could capture analogies like "king − man + woman ≈ queen."
The utility of embeddings extends far beyond natural language processing. Graph embeddings encode node relationships in social or knowledge networks. Item embeddings power collaborative filtering in recommendation systems. Vision models produce image embeddings that enable similarity search across millions of photos. In each case, the embedding serves the same function: compressing complex relational structure into a form that downstream models can efficiently consume. Pretrained embeddings can also be transferred across tasks, allowing a representation learned on one large dataset to accelerate learning on a smaller, related problem.
Embeddings have become one of the most universally applicable tools in modern machine learning. They underpin large language models, multimodal systems, and retrieval-augmented generation pipelines, where dense vector search over embedding spaces enables fast, semantically meaningful lookup. As models grow larger and more capable, the quality and structure of their internal embeddings have become a primary lens through which researchers interpret what these systems have learned about the world.
