Adapter

Adapters are lightweight, modular components inserted into a pre-trained neural network to enable task-specific fine-tuning without modifying the original model's parameters. Rather than updating all weights during fine-tuning — a process that is computationally expensive and requires storing a separate full model copy per task — adapters introduce small bottleneck layers or modules at strategic points within the network architecture. Only these adapter parameters are trained on the downstream task, while the backbone model remains frozen. This design dramatically reduces the number of trainable parameters, often to less than 1% of the original model's total weights.

In practice, adapters are typically inserted between the layers of a transformer-based model. A common design uses a down-projection matrix to compress the hidden representation into a lower-dimensional space, applies a nonlinearity, then uses an up-projection to restore the original dimensionality — with a residual connection bypassing the module. This bottleneck structure keeps the adapter small while still giving the model enough expressive capacity to adapt to new tasks. The original model weights are shared across all tasks, and only the task-specific adapter weights need to be swapped out at inference time.

Adapters matter because they make large pre-trained models practical to deploy across many tasks simultaneously. In a traditional fine-tuning paradigm, serving ten tasks would require ten full copies of a large model. With adapters, a single shared backbone can serve all tasks by loading only the relevant adapter weights — a significant reduction in storage and memory overhead. This modularity also enables continual learning scenarios where new tasks can be added without risking catastrophic forgetting of previously learned ones.

The approach gained traction in NLP following the 2019 paper "Parameter-Efficient Transfer Learning for NLP" by Houlsby et al., which demonstrated competitive performance on GLUE benchmarks using adapters with BERT while training only a small fraction of total parameters. Since then, adapter-based methods have expanded into computer vision, multimodal models, and speech, and have inspired a broader family of parameter-efficient fine-tuning (PEFT) techniques including LoRA, prefix tuning, and prompt tuning.