Mid-Training

Mid-training is an intermediate stage in the modern large-language-model training pipeline, sitting between broad, large-scale pretraining and post-training (supervised fine-tuning, preference optimization, or reinforcement learning). The model keeps training with a next-token or next-token-like objective, but on a smaller, more curated or synthetic data mixture than the original pretraining corpus, often while the learning rate anneals toward zero. It typically consumes a few percent to low double-digit percent of total pretraining compute: several hundred billion tokens, compared with tens of trillions during pretraining.

The term traces to 2024 model reports, including Microsoft's Phi team ("Phi-4 Technical Report," Abdin et al., 2024) and OpenAI. The Allen Institute for AI adopted it explicitly in the OLMo 2 report (arXiv:2501.00656), describing a "Stage 2" that anneals the learning rate on a curated data mix called Dolmino. Mid-training is used to patch capability gaps that survive pretraining, such as weak math or code performance, to extend context length, and to stabilize convergence near favorable minima. Supervised fine-tuning works differently: it trains on chat-formatted instruction-response pairs, not raw corpora.

A 2025 survey, "Mid-Training of Large Language Models: A Survey" (arXiv:2510.06826), organizes the practice into data-distribution shifts, learning-rate scheduling, and long-context extension. It explains why the approach works using gradient-noise-scale, information-bottleneck, and curriculum-learning arguments. By 2026 the term appears widely across model reports and research on how mid-training interacts with later reinforcement learning and distillation stages, including its role in tradeoffs between knowledge retention and reasoning.