Data Mixture
The composition of training data — the proportions of different sources, domains, languages, and document types — used to train a machine learning model.
A data mixture is the composition of a model's training corpus: the relative proportions of different data sources, domains (web, books, code, scientific papers, dialogue), languages, document types, and quality strata that are mixed together before or during pretraining. Data mixture is one of the most consequential design choices in modern pretraining pipelines — small changes in mixture proportions can shift a model's downstream capabilities substantially, and frontier model labs treat the recipe as proprietary. Mixture design involves tradeoffs between coverage (diversity of topics and languages), quality (weighting cleaner sources like curated textbooks over noisy web scrapes), and capability transfer (some data sources disproportionately improve downstream reasoning). Data mixture inference is the related but distinct practice of reverse-engineering a closed model's mixture from its external behavior. Public mixture specifications (e.g., the GPT-3, LLaMA, and DeepSeek papers) have become a reference grammar for the field even when exact ratios are not reproduced.
Sources
- Exploring Claude/GPT Knowledge Cutoffs & Pre-training Timelines
Shrivu's Substack · Aug 10, 2026
- Language Models are Few-Shot Learners
arXiv (NeurIPS 2020) · May 28, 2020
- LLaMA: Open and Efficient Foundation Language Models
arXiv · Feb 27, 2023
Research this in Signals
Signals turns a topic into a sourced research record you can inspect and rerun. Your first scan is free, and this one starts with Data Mixture already loaded, so edit it or scan as is.