Distillation Tax
Performance ceiling when training smaller models from larger model outputs
Distillation Tax refers to the performance gap between a student model trained through knowledge distillation and the teacher model from which it learns. Knowledge distillation trains a small model to mimic a large one and is a common approach for creating deployable models. Empirical evidence shows that student models consistently underperform their teachers, and the gap can persist regardless of training technique or data quantity. This gap is the distillation tax. It is a structural limitation: the student inherits the teacher's knowledge along with its blind spots, failure modes, and implicit biases.
The student learns a compressed approximation of the teacher's behavior. The teacher encodes capabilities and knowledge in its weights; distillation attempts to transfer this through soft targets (probability distributions) and matching hidden states. The student has fewer parameters, lower capacity, and must make tradeoffs about what to memorize versus what to generalize. When the teacher makes a systematic error or has a particular weakness, the student often amplifies it. The student also cannot invent knowledge the teacher lacks; it is upper-bounded by the teacher's competence. Even with perfect knowledge transfer, the student's smaller capacity prevents it from expressing everything the teacher knows. A 7B model trained on outputs from a 70B teacher will have a performance ceiling below the teacher, no matter how careful the distillation process.
The tax matters for AI development strategy. As models scale up, there is pressure to create smaller, cheaper, faster versions. Distillation is a key technique for this, yet the tax means efficiency comes at a cost. Organizations must decide whether a distilled model is sufficient for their use case or whether they need the full capability. The existence of a fundamental tax also points to a limitation of pure distillation; capable, efficient models may require alternatives such as architecture innovation, different training paradigms, or accepting the tradeoff as the cost of deployment.
Research this in Signals
Signals turns a topic into a sourced research record you can inspect and rerun. Your first scan is free, and this one starts with Distillation Tax already loaded, so edit it or scan as is.
