Skip to content

LearnRank & assemble

Cross-encoder

A model that reads a query and a passage together in one pass and outputs a single relevance score for the pair.

In DocSlurpIllustrated explainer

On this page
Bi-encoder versus cross-encoderTop, a bi-encoder: the query "how fast get Learjet london" and the passage "wheels-up within three hours of request" go through separate encoders. Attention links stay inside each text. Each side becomes one vector; the passage vector is computed once at ingest and stored. The two vectors meet at a dot product, giving an illustrative similarity of 0.41. Bottom, a cross-encoder: one input, CLS, the query, SEP, the passage, SEP, goes through one transformer. Attention links cross between the texts, for example fast to within, Learjet to wheels-up, how to three and fast to hours. The output is one relevance score, illustratively 0.94. Nothing can be precomputed, so each query and passage pair needs its own model pass.Bi-encodertwo separate passes, compared at the endhowfastgetLearjetlondonwheels-upwithinthreehoursofrequestEncoderEncoder (run once, at ingest)query vectorstored in the index·similarity 0.41The passage was summarized before the question existed. A query costs one model pass plus a vector lookup.Cross-encoderone pass over both texts together[CLS]howfastgetLearjetlondon[SEP]wheels-upwithinthreehoursofrequest[SEP]One transformer reads both texts: every token can attend to every other tokenrelevance 0.94Nothing can be computed ahead: every query and passage pair is a fresh model pass, so it runs on a shortlist.
A bi-encoder compares two summaries; a cross-encoder reads the pair. In the bi-encoder, attention stays inside each text, and the passage was squeezed into one vector at ingest, before anyone asked. In the cross-encoder, “fast” can attend to “within” and “Learjet” to “wheels-up”, so the model can tell that this passage answers the question even with no words in common. The price: no precomputation. Words stand in for subword tokens; scores are illustrative.

The problem: the passage was summarized before the question existed

Vector search uses a bi-encoder. At ingest, a model turns each passage into one vector and stores it. At query time, the same kind of model turns your question into a vector, and the search compares the two with a dot product or cosine similarity.

Take the question “How fast can my assistant get the Learjet to London?” and the passage “Wheels-up within three hours of the request.” They share no words. Whether this pair scores well depends on whether the passage vector, computed months ago, happened to land near the question vector. That one vector had to stand in for every question the passage could ever answer. It can’t encode how the passage relates to this question, because it was built before the question existed.

Reading the pair together

A cross-encoder skips the separate summaries. It joins the question and passage into one input:

[CLS] how fast can my assistant get the learjet to london [SEP] wheels-up within three hours of the request [SEP]

A transformer reads the whole thing, and a small output layer turns it into one number: how well this passage answers this question. Inside the model, attention lets every token look at every other token, including across the separator. “Fast” can attend to “within three hours”, “Learjet” to “wheels-up”. The model judges the relationship directly instead of comparing two summaries.

That is why cross-encoders catch things bi-encoders blur:

  • Negation and conditions. “The jet does not leave until the CFO approves” looks almost the same as the positive version to a single vector. A model reading the pair can see the “not”.
  • Which thing is meant. A question about the Learjet’s flight time versus its charter cost lands near the same vectors. Read side by side, the difference is obvious.
  • Numbers and units. “Three hours” versus “seven hours” matters to the question, and a joint reading can weigh it.

The cost

Nothing can be computed ahead of time, because the input is the pair. Work it through for a corpus of 1,000,000 passages:

Approach Model passes per query
Bi-encoder 1 (embed the question), plus a vector index lookup
Cross-encoder over everything 1,000,000
Cross-encoder over a 200-passage shortlist 200

The shortlist row is 5,000 times fewer passes than scoring everything. This is why cross-encoders are almost always used for reranking: a bi-encoder or BM25 finds 200 plausible passages quickly, and the cross-encoder reorders only those.

Late interaction models such as ColBERT sit between the two. They store one vector per token at ingest and compare token against token at query time. You get some word-level matching while keeping the passage side precomputed, at the price of a much larger index.

How they are trained and what the score means

Cross-encoders are trained on (question, passage, label) examples, often from large search datasets such as MS MARCO, to give relevant pairs high scores and irrelevant ones low scores. The output is a raw number from the model. Some models pass it through a sigmoid to land between 0 and 1. Even then it is a ranking signal for one model, not a calibrated probability, and scores from two different rerankers are not comparable.

Newer rerankers are built from small language models. The Qwen3 reranker, for example, is prompted to judge whether a document meets the query, and its score comes from how strongly it leans toward “yes” over “no”. The idea is the same: one model, both texts, one score.

Where it breaks

  • Cost is linear in candidates. Every candidate is a full model pass. Deep shortlists or long passages make latency climb fast.
  • Input length is capped. Question plus passage must fit the model’s input. Long passages get truncated, and the key sentence may be in the part that was cut.
  • It only reorders. A cross-encoder can’t find a passage the first stage didn’t return.
  • Domain shift. A model trained on web search may misjudge tables, transcripts or legal clauses. Evaluate it on your own questions.

In DocSlurp

DocSlurp’s default reranker is not a cross-encoder. It is BM25 over the candidate pool (core/server/src/search/rerank.ts), and it stores its result in a field named crossScore despite that. A stub called rerankRemote in that file also just runs BM25.

The cross-encoder style option is the hosted Qwen/Qwen3-Reranker-0.6B (core/server/src/search/hosted-rerank.ts). A project owner has to opt in. It then scores up to 200 candidates in batches of 25, checks the project privacy mode and spend allowance before each batch, records token usage, and falls back to BM25 if anything fails. See retrieval controls.

When to use one

Use a cross-encoder when the top few results decide the outcome, as in RAG, and when you can spend tens to hundreds of model passes per query. Pair it with a fast first stage that has good recall. If you can’t afford a model call per candidate, a bi-encoder with hybrid search is the cheaper baseline to beat.

In DocSlurp

In DocSlurp

Optional: a project can select the hosted Qwen/Qwen3-Reranker-0.6B, which scores query and passage together, with privacy and spend checks and a BM25 fallback (core/server/src/search/hosted-rerank.ts). It is off unless a project owner opts in. The default reranker stores its result in a field named crossScore, but it is BM25, not a cross-encoder (core/server/src/search/rerank.ts).

Sources

Jump to