Cross-Encoder

Cross-encoder

A cross-encoder is a model that takes a query and a candidate document as a single input, encodes them together, and outputs a relevance score. The architecture is the opposite of a dual-encoder (also called bi-encoder): a dual-encoder encodes the query and document separately into independent vectors and compares them with a cheap similarity function, while a cross-encoder lets the query and document attend to each other through every transformer layer before producing the score. Cross-attention between query and document captures fine-grained relevance signals that independent encoders cannot, which is why cross-encoders tend to score more accurately. Every (query, document) pair requires a full forward pass through the model, which makes cross-encoders impractical for first-stage retrieval over large corpora. Cross-encoders are therefore typically used as second-stage rerankers on a small shortlist produced by a cheaper retriever. They are the architecture behind most modern rerankers, including Cohere Rerank, the monoT5 / monoBERT family, and many LLM-based relevance judges.

Mechanically, a cross-encoder is a transformer (typically BERT-family or RoBERTa-family, sometimes a small LLM) trained on (query, document, relevance label) triples. The input is the concatenation of the query, a separator token, and the document. The model's [CLS] token (or equivalent pooled output) is passed through a classification head that emits a score, usually binary (relevant / not relevant) or graded (e.g., 0–4). Training uses cross-entropy loss on labeled relevance data, often assembled from click logs, human annotations, or synthetic judgments. At inference, scoring N candidates for a query is N forward passes, not one, which is the dominant cost. Efficiency variants (ColBERT's late-interaction, ColBERTv2, the monoT5 family) precompute document representations and only score the query against them, trading some accuracy for speed. These variants behave like cross-encoders in spirit (rich interaction) without paying the full per-pair cost.

The cross-encoder frame is useful because it names the architecture that delivers the highest retrieval quality per candidate scored, and it explains why rerankers sit as a separate stage in retrieval pipelines. The frame also clarifies the scaling limits of retrieval: a corpus of N documents requires N × M forward passes to rerank M queries with a cross-encoder, and that cost grows linearly with both N and M. A limitation is that the frame is often described in contrast to dual-encoders without acknowledging the hybrid middle ground, where late-interaction, multi-vector, and SPLADE-style sparse representations sit between the two. There is also a real difference between cross-encoders and LLM-based relevance judges. An LLM-as-judge is essentially a cross-encoder with a generative interface, but it carries the calibration and verbosity issues that LLMs have, while a trained cross-encoder behaves closer to a calibrated classifier.

Open questions include how to reduce the per-pair inference cost of cross-encoders without losing their accuracy advantage. Late-interaction methods are the current answer, but the tradeoff frontier is still moving. Another question is how to make cross-encoders robust to distribution shift when the relevance distribution changes, such as a new domain or new query types. There is also a question about whether LLM-based judges will replace trained cross-encoders in practice, given that LLMs are more flexible but less calibrated. The deeper question is how to combine retrieval and reasoning when the relevance signal depends not just on lexical and semantic overlap but on whether the document supports the user's actual task, and this remains an open problem.