Switch Distillation

Switch Distillation is a training objective introduced by Jacqueline He and coauthors from the University of Washington, Princeton Language and Intelligence, and Meta AI in "Knowledge Distillation During Mid-Training Favors Reasoning over Factual Recall" (arXiv:2609.01532, September 2026). It addresses a tradeoff the paper identifies in standard logit-based knowledge distillation during mid-training. Relative to plain next-token prediction (NTP), forward-KL distillation from a stronger teacher improves reasoning but slows the acquisition of low-entropy factual knowledge, because teacher models are more confident on procedural data than on knowledge-intensive data.

Switch Distillation resolves this by using the teacher's predictive entropy as a per-token routing signal. On tokens where the teacher is confident (low entropy), the student trains with the standard distillation loss. On tokens where the teacher is uncertain, the objective falls back to ordinary cross-entropy against the ground-truth token. This keeps the reasoning gains of distillation on data where the teacher's soft targets are informative, while protecting factual acquisition on data where they are not.

In the paper's experiments, Switch Distillation outperforms both plain NTP and standard forward or reverse KL distillation across teacher sizes. It reaches 1.61 to 1.71 times the reasoning performance and 1.13 to 1.19 times the knowledge and commonsense performance of NTP, while preserving 96.7 to 96.8% of factual recall. These gains persist through subsequent post-training (supervised fine-tuning, preference optimization, and reinforcement learning with verifiable rewards). By that point, the factual-recall gap closes entirely.