In the context of language modeling, perplexity is essentially a measurement of how "surprised" a model is by a given sequence of words. It is calculated as the exponential of the average negative log-likelihood of a sequence of words, given the model. Lower perplexity indicates that the model is more confident in its predictions, suggesting better performance. This metric is particularly useful in comparing different models or tuning models for optimal performance in tasks like speech recognition, machine translation, and other natural language processing (NLP) applications. Perplexity serves not only as a benchmark for model evaluation but also as a guiding metric in the development of more efficient and accurate language models.
The concept of perplexity has been used in statistics and information theory long before its adoption in language modeling, but it gained prominence in the AI field in the late 20th century, particularly with the rise of statistical methods in NLP in the 1980s and 1990s.
While the adaptation of perplexity to language modeling does not attribute to a single individual, it has been extensively developed and refined by researchers in the field of computational linguistics and natural language processing. Organizations like IBM's research divisions played a significant role in its early development and application to machine translation and speech processing systems.