In AI, particularly in fields like deep learning and natural language processing, similarity masking is employed to emphasize or suppress certain elements of data based on their similarity scores. This approach helps models focus on critical features by "masking" or reducing the impact of redundant or overly similar data points, leading to improved model efficiency and accuracy. For instance, in attention mechanisms (such as those used in transformers), similarity masking can prevent a model from giving undue attention to inputs that are too similar, ensuring diversity in the information processed. It is especially useful in complex datasets where certain patterns may dominate and hinder learning from more subtle, yet significant, distinctions.
The concept of masking in AI has been part of broader efforts in feature selection and dimensionality reduction since the 1990s. However, similarity masking, in particular, gained attention with the development of attention mechanisms in the mid-2010s, particularly with the advent of transformer models in 2017.
The introduction and popularization of similarity masking are closely tied to the researchers behind the transformer architecture, especially Vaswani et al., who introduced the transformer in 2017. Their work on attention mechanisms spurred the development of various masking techniques used in modern AI.