---
title: Data Mixture Inference
type: vocabulary
url: "https://www.envisioning.com/vocab/data-mixture-inference"
summary: "Reverse-engineering the composition of a model's training corpus from its observable behavior, particularly tokenization artifacts and output patterns."
year: 2024
generality: 0.65
---

# Data Mixture Inference

Reverse-engineering the composition of a model's training corpus from its observable behavior, particularly tokenization artifacts and output patterns.
Data mixture inference is the practice of estimating the composition of a model's pretraining corpus — the relative proportions of different data sources, domains, and languages — by observing the model's external behavior. Techniques include analyzing tokenization patterns, measuring perplexity on held-out text from different sources, and probing how the model breaks down or reproduces specific phrasings. The technique is used both by researchers studying foundation model training pipelines and by adversarial actors seeking to extract proprietary information about closed models. Data mixture inference differs from membership inference attacks (which target individual examples) in that it targets aggregate distributional properties of the training set.

---
Source: Envisioning — Technology Research Institute (https://www.envisioning.com/vocab/data-mixture-inference)