TAID (Temporally Adaptive Interpolated Distillation)
A distillation technique that aligns teacher and student models across differing temporal resolutions.
Temporally Adaptive Interpolated Distillation (TAID) is a knowledge distillation framework designed specifically for sequence and temporal models — such as those processing video, speech, or sensor streams — where the teacher and student operate at different temporal resolutions or under different latency constraints. Rather than applying distillation losses directly on misaligned timesteps, TAID interpolates the teacher's representations, soft labels, or feature maps onto the student's coarser temporal grid, enabling meaningful supervision even when the two models sample time at fundamentally different rates.
The core mechanism involves aligning teacher and student sequences through interpolation strategies ranging from simple linear or spline methods to learned temporal attention kernels that can adapt to the structure of the data. Once aligned, distillation losses are applied across multiple signal types: per-timestep feature regression encourages the student to mimic intermediate teacher representations, temporally smoothed KL divergence on output logits transfers predictive distributions, and continuity regularizers preserve the dynamic structure of the sequence rather than treating each frame independently. A key innovation is the temporally adaptive weighting scheme, which concentrates distillation pressure on informationally dense moments — motion boundaries in video, phoneme transitions in speech — while downweighting redundant or static frames. This focus makes the compressed student model more robust to frame-rate variation and better at capturing fine-grained temporal patterns despite operating on subsampled inputs.
TAID addresses a practical bottleneck in deploying sequence models at scale: high-performing teachers are often trained with dense temporal sampling and large receptive fields, while real-world deployment demands low-latency, low-compute students that cannot afford the same resolution. Applications span action recognition, temporal action segmentation, online event detection, streaming automatic speech recognition, and efficient sensor-based inference. The approach sits at the intersection of knowledge distillation, temporal alignment theory, and sequence modeling, building on foundational distillation work and intermediate representation transfer methods like FitNets while extending them into the temporal domain.
TAID emerged in the early 2020s as research and industry increasingly prioritized real-time sequence model compression, with the framework gaining broader recognition around 2022–2024 as frame-rate-robust and streaming-capable distillation became a recognized subfield within efficient deep learning.
