Data Mixture Inference

Data mixture inference is the practice of estimating the composition of a model's pretraining corpus — the relative proportions of different data sources, domains, and languages — by observing the model's external behavior. Techniques include analyzing tokenization patterns, measuring perplexity on held-out text from different sources, and probing how the model breaks down or reproduces specific phrasings. The technique is used both by researchers studying foundation model training pipelines and by adversarial actors seeking to extract proprietary information about closed models. Data mixture inference differs from membership inference attacks (which target individual examples) in that it targets aggregate distributional properties of the training set.