Self-supervised learning (SSL) represents a paradigm in machine learning that enables models to learn useful representations of data without explicit labels. This technique often involves creating a predictive task where the input itself serves as its own label, thus eliminating the need for human-annotated data. Common methods in SSL include predicting missing parts of data, solving puzzles derived from the data itself, or forecasting future elements in sequences. The power of SSL lies in its ability to leverage large volumes of unlabeled data, making it particularly useful in domains where labeled data is scarce or expensive to obtain. It has been especially impactful in fields such as natural language processing and computer vision, where it has significantly advanced the state-of-the-art by enabling models to learn more generalizable and robust features.
The concept of self-supervised learning has been discussed in various forms since the early 2000s, but it gained significant traction around the mid-2010s with the rise of deep learning. Its popularity soared as researchers discovered its effectiveness in leveraging unlabeled data to improve learning algorithms' performance.
Key figures in the development and popularization of self-supervised learning include Yann LeCun, who has been a strong advocate for the approach, describing it as part of the "cake" of machine learning methods. Other notable contributors include researchers at major AI labs like Google Brain and Facebook AI Research (FAIR), who have developed several innovative SSL techniques and applications.