Pretrained models serve as a powerful starting point for various machine learning and deep learning tasks, capitalizing on the knowledge (weights and biases) obtained from training on extensive datasets. These models significantly reduce the computational cost and time required for training, as they can be fine-tuned with a relatively small amount of data to perform specific tasks. This approach is particularly beneficial in domains where data is scarce or when computational resources are limited. Pretrained models are widely used in natural language processing (NLP), computer vision (CV), and other areas, enabling state-of-the-art performances in tasks such as image recognition, language translation, and voice recognition. By leveraging transfer learning, these models can adapt to new, but related, problems by adjusting the pre-learned representations to the specifics of the new task.
The concept of pretrained models gained significant popularity in the mid-2010s, especially with the rise of deep learning networks like Convolutional Neural Networks (CNNs) for image tasks and Transformer models for NLP tasks.
While it's challenging to attribute the concept of pretrained models to specific individuals, organizations like Google, OpenAI, and Facebook's AI Research (FAIR) lab have been instrumental in advancing and popularizing the use of pretrained models through the release of models like BERT (Bidirectional Encoder Representations from Transformers) for NLP and ResNet for computer vision tasks.