Self-Distillation

Self-distillation is a variant of knowledge distillation in which the student model has the same architecture as the teacher, rather than a smaller one. In standard distillation a compact student is trained to match the outputs of a larger, more capable teacher, mainly to compress a model. Self-distillation drops the size gap: a network is retrained to match the soft predictions of an earlier copy of itself, sometimes over several rounds. Furlanello et al. named this pattern Born-Again Networks in 2018 (arXiv:1805.04770). They showed that identically sized students trained this way could outperform their own teacher on image and language benchmarks, a result that's counterintuitive since no new labeled data enters the loop.

A related form applies the idea within a single forward pass rather than across separate training runs. Shallow classifiers attached partway through a deep network are trained to match the predictions of the network's own final layers, letting early layers benefit from supervision that would otherwise only reach the output (Zhang et al., 2019, arXiv:1905.08094). Both forms have been studied theoretically. A 2020 NeurIPS analysis showed that, in a simplified functional-regression setting, repeated self-distillation acts as a form of regularization, progressively restricting the basis functions available to the model. Later work found the resulting minima are flatter in the loss landscape, which is one proposed explanation for why self-distilled models often generalize better on held-out data.

Self-distillation needs no separate large teacher, no additional labels, and no architecture change. That makes it a low-cost addition to existing training pipelines: it's used to squeeze extra accuracy out of a fixed model rather than to compress one, and to build ensembles or multi-round training and reasoning-refinement loops in more recent LLM work.