Model Stitching

Model stitching is a technique that connects the early layers of one pretrained model to the late layers of another using learned transformation layers, allowing capabilities to be transferred between models with different architectures without fine-tuning the frozen components.

The stitched model takes representations from the source model's early backbone — which captures fundamental features like syntax and semantics — and maps them through a lightweight learned adapter into the target model's decoder head for a specific task. This enables reusing the frozen knowledge encoded in a large foundation model without the computational cost of running its entire forward pass.

Research has shown that the middle layers of large language models are relatively architecture-agnostic, meaning a simple linear projection often suffices to bridge between different model families. A projection learned on one model pair often generalizes to other pairs, suggesting the representational space is more universal than the architecture.

Model stitching can combine the language understanding of one model with the formatting preferences of another, all without fine-tuning the frozen components. This makes it possible to customize model behavior — switching output styles, adding domain knowledge, or adapting to new tasks — by only training the thin stitching layer.