Knowledge Distillation

Knowledge distillation

Knowledge distillation transfers capabilities from a large, high-performing model to a smaller, more efficient model by training the student on the teacher's output distributions rather than raw labels.

The student learns from the full probability distribution over all possible outputs that the teacher produces instead of one-hot correct answers. For example, an image of a cat might receive a probability of 0.7 for the cat class and 0.25 for the dog class. The student learns the significance of these relative probabilities. The soft targets capture dark knowledge: relationships between classes and confidence calibration that plain labels cannot express. Modern approaches often combine temperature-scaled softmax outputs with contrastive learning signals to preserve the teacher's predictions and learned representations.

Distilled models can achieve 90–95% of the teacher's performance while requiring substantially less compute and memory. Performance gaps remain for rare or ambiguous cases in which the teacher's probability distributions are themselves uninformative. The teacher must be substantially larger and more capable than the student for the knowledge transfer to be meaningful, and the student inherits any biases in the teacher.

The theoretical conditions under which dark knowledge is maximally useful for a student have not been fully characterized. Whether factual, procedural, or stylistic knowledge distills more effectively than other types remains an open empirical question. Reliable scaling laws have not yet captured the relationship between teacher-student capacity ratio and downstream task performance.