Encoder-Free Early Fusion

Encoder-free early fusion is an architectural approach in which raw audio and video signals are processed without large, pre-trained standalone encoders — such as a Whisper-like automatic speech recognition (ASR) model for audio or a CNN-based vision encoder for images. Instead, audio signals are passed through a lightweight embedding layer (for example, converting to dMel spectrogram representations), and images are split into fixed-size patches encoded by small MLPs (hMLPs), with the entire system — embedding layers, encoders, and core transformer — co-trained end-to-end from scratch.

The contrast is with the dominant paradigm in multimodal AI, where large, separately pre-trained encoders are frozen or fine-tuned and their outputs are fused at a late stage. A typical production system might use Whisper for speech-to-text, a CLIP-style vision encoder for images, and a language model for reasoning — three separately trained models stitched together. The encoder-free early fusion approach replaces this assembly with a single jointly trained system that processes raw modality inputs from the start.

The motivation is the bitter lesson (Sutton 2019): hand-crafted components tend to be outpaced by learned components as scale increases. An ASR encoder pre-trained on speech data separately from the core reasoning model represents a hand-crafted information bottleneck — it decides what information to preserve from the audio signal before the reasoning model has any opportunity to influence that decision. By co-training the entire system end-to-end, the embedding layers learn to preserve whatever information the reasoning task ultimately requires, without the information loss that comes from early commitment to a fixed representation.

The cost is training complexity. End-to-end co-training of a system that processes audio, video, and text from raw inputs requires substantially more compute and careful stabilization work than assembling pre-trained components. The encoder-free approach also sacrifices the generalization benefits of pre-training on massive speech and image datasets, meaning it requires sufficient co-training data to learn good representations from scratch. Whether the quality gains from native end-to-end processing outweigh these costs is an empirical question that depends on dataset size, compute budget, and the difficulty of the target tasks.