Cross-Encoder

A cross-encoder is a model that takes a query and a candidate document as a single input, encodes them together, and outputs a relevance score. The architecture is the opposite of a dual-encoder (also called bi-encoder): where a dual-encoder encodes the query and document separately into independent vectors and compares them with a cheap similarity function, a cross-encoder lets the query and document attend to each other through every transformer layer before producing the score. The advantage is accuracy: cross-attention between query and document captures fine-grained relevance signals that independent encoders cannot. The cost is computational: every (query, document) pair requires a full forward pass through the model, which makes cross-encoders impractical for first-stage retrieval over large corpora. Cross-encoders are therefore typically used as second-stage rerankers on a small shortlist produced by a cheaper retriever. They are the architecture behind most modern rerankers, including Cohere Rerank, the monoT5 / monoBERT family, and many LLM-based relevance judges.

Mechanically, a cross-encoder is a transformer (typically BERT-family or RoBERTa-family, sometimes a small LLM) trained on (query, document, relevance label) triples. The input is the concatenation of the query, a separator token, and the document; the model's [CLS] token (or equivalent pooled output) is passed through a classification head that emits a score — usually binary (relevant / not relevant) or graded (e.g., 0–4). Training uses cross-entropy loss on labeled relevance data, often assembled from click logs, human annotations, or synthetic judgments. At inference, scoring N candidates for a query is N forward passes, not one, which is the dominant cost. There are efficiency variants (ColBERT's late-interaction, ColBERTv2, the monoT5 family) that precompute document representations and only score the query against them, trading some accuracy for speed — these are conceptually cross-encoders in spirit (rich interaction) without paying the full per-pair cost.

The advantage of the cross-encoder frame is that it names the architecture that delivers the highest retrieval quality per candidate scored, and explains why rerankers are a separate stage in retrieval pipelines. The frame is also useful for understanding the scaling limits of retrieval: a corpus of N documents requires N × M forward passes to rerank M queries with a cross-encoder, and that cost grows linearly with both N and M. The cost of the frame is that it is often described in contrast to dual-encoder without acknowledging the hybrid middle ground (late-interaction, multi-vector, and SPLADE-style sparse representations sit between the two). There is also a real difference between cross-encoders and LLM-based relevance judges: an LLM-as-judge is essentially a cross-encoder with a generative interface, but it brings all the calibration and verbosity issues that LLMs have, while a trained cross-encoder is closer to a calibrated classifier.

Open questions include how to reduce the per-pair inference cost of cross-encoders without losing their accuracy advantage — late-interaction methods are the current answer, but the tradeoff frontier is still moving — and how to make cross-encoders robust to distribution shift when the relevance distribution changes (e.g., a new domain, new query types). There is also an open question about whether LLM-based judges will replace trained cross-encoders in practice, given that LLMs are more flexible but less calibrated. The deeper question — how to combine retrieval and reasoning when the relevance signal depends not just on lexical and semantic overlap but on whether the document supports the user's actual task — is one of the open problems the field is working on.