In the context of machine learning, particularly in deep learning, the Hierarchy of Generalizations refers to the way neural networks, especially deep neural networks, learn to represent data. Lower layers of a neural network typically capture fine-grained, specific features (like edges in an image), while higher layers combine these features into more abstract and generalized representations (like shapes or objects). This hierarchical structure is crucial for the network’s ability to generalize from the training data to unseen data, allowing it to perform tasks like image recognition, natural language processing, or game playing with high accuracy. The idea is that as you move up the hierarchy, the learned features become increasingly invariant to irrelevant variations in the data, making them more useful for high-level tasks.
The concept of hierarchical feature extraction can be traced back to early models of the visual cortex in the 1960s, but it became more formalized in machine learning with the advent of deep learning in the early 2000s, particularly after the resurgence of neural networks in 2006 with Hinton et al.'s work on deep belief networks. The term "Hierarchy of Generalizations" gained traction as deep learning models, like convolutional neural networks (CNNs), demonstrated success in various domains.
Key contributors to the development of this concept include Geoffrey Hinton, Yoshua Bengio, and Yann LeCun. Hinton's work on deep belief networks and LeCun's development of convolutional neural networks were instrumental in establishing the hierarchical approach to learning in neural networks. Their contributions laid the groundwork for understanding how neural networks can automatically discover and utilize hierarchies of features in data.