Late interaction (ColBERT)
ColBERT · MaxSim · Multi-vector retrieval
A retrieval method that keeps one vector per token and scores a passage by matching each query token to its best passage token, then summing those best matches.
Related techniqueIllustrated explainer
On this page
The problem: one arrow per passage loses detail
A standard bi-encoder squeezes a whole passage into a single vector. For “The travel desk reserves the Learjet charter, then arranges catering and a car at Farnborough”, the model produces a vector for every token and then averages them into one. That average has to stand for everything at once: reservations, aircraft, catering, cars, airports. When the question is “How does booking the jet to London work?”, the single vector can still match, but the specific evidence, “charter” answering “jet” and “reserves” answering “booking”, has been blended away before the comparison happens.
The alternative at the other extreme is a cross-encoder, which reads the question and the passage together and catches every detail, but has to run the full model once per candidate at query time. That is far too slow for a whole corpus.
Late interaction sits between them. Encode the query and the passage separately, as a bi-encoder does, so passages can still be processed at ingest. But keep one vector per token instead of pooling, and let the two sides interact only at the very end, with a cheap operation.
How MaxSim works
ColBERT, the model that introduced the idea, scores a passage like this:
- Embed each query token into a vector, and each passage token into a vector. Normalize them.
- For every query token, compute its dot product with every passage token. That is a grid: query tokens down the side, passage tokens across the top.
- In each row, keep only the maximum. That is the best match for that query token anywhere in the passage.
- Add up the row maxima. That sum is the passage’s score.
score(q, d) = Σ over query tokens i max over passage tokens j (qᵢ · dⱼ)
That’s why it is called MaxSim: a sum of maximum similarities.
Work it through
Take the query tokens jet, london, booking and two short passages. The cell values are illustrative dot products between unit token vectors, chosen to be easy to check.
travel-desk-guide.pdf: “desk reserves a charter london”
| desk | reserves | a | charter | london | row max | |
|---|---|---|---|---|---|---|
| jet | 0.21 | 0.10 | 0.05 | 0.78 | 0.40 | 0.78 |
| london | 0.12 | 0.08 | 0.04 | 0.35 | 0.97 | 0.97 |
| booking | 0.30 | 0.74 | 0.06 | 0.15 | 0.22 | 0.74 |
Score: 0.78 + 0.97 + 0.74 = 2.49.
growth-plan-2027.pdf: “revenue growth per london quarter”
| revenue | growth | per | london | quarter | row max | |
|---|---|---|---|---|---|---|
| jet | 0.25 | 0.09 | 0.03 | 0.40 | 0.08 | 0.40 |
| london | 0.10 | 0.06 | 0.04 | 0.97 | 0.12 | 0.97 |
| booking | 0.31 | 0.12 | 0.05 | 0.22 | 0.10 | 0.31 |
Score: 0.40 + 0.97 + 0.31 = 1.68.
Both passages contain “london”, so that row is a tie. The travel desk guide wins because each remaining query word found its own partner: “charter” for “jet”, “reserves” for “booking”. The growth plan has nothing for them, and the best it can offer is a weak match. Each query token has its say, and no single strong match can carry an otherwise irrelevant passage.
The token vectors are contextual, which matters. The vector for “charter” was computed while reading the whole passage, so it already carries “charter aircraft” rather than “charter” as in a company’s founding document. That is what lets a token-level comparison capture meaning, not just spelling.
What it costs
The price is storage. A 200-token passage stores 200 vectors instead of one. ColBERT keeps token vectors small (128 dimensions in the original paper), and ColBERTv2 compresses each one aggressively, but a late-interaction index is still many times larger than a single-vector index over the same text.
Query time is cheap per comparison but involves many comparisons: a 10-token query against a 200-token passage is 2,000 dot products instead of one. Systems keep this practical with a first stage that finds candidates using token-level approximate search, then compute exact MaxSim only for those.
The same idea works for page images. ColPali embeds a page screenshot as a grid of patch vectors and scores it against query tokens with MaxSim, so a question can match the one table cell or chart label that answers it.
Where it breaks
- Storage and memory. The index grows with the number of tokens, not the number of passages. For a large corpus that is the deciding cost.
- Every query token counts equally. Words like “the” or “does” also take a row maximum. ColBERT handles this partly through training and by filtering punctuation, but a long rambling query adds noise to every score.
- It still can’t read across the pair. Late interaction is closer to a cross-encoder than a bi-encoder, but query and passage tokens never attend to each other. Questions that hinge on how two facts combine still benefit from a cross-encoder reranker.
- Tooling is less common. Most vector databases are built around one vector per row. Multi-vector search needs specific index support.
When to reach for it
Late interaction pays off when single-vector search keeps returning passages that are about the right topic but miss the specific detail, and when a cross-encoder over a deep candidate list is too slow or expensive. It is also a strong option for visually rich documents through ColPali-style models. If storage is tight or your corpus is huge, a single-vector first stage plus a reranker over the shortlist is the usual cheaper alternative.
In DocSlurp
Related technique
DocSlurp does not implement late interaction. Text search stores one vector per chunk (core/worker/src/stages/embed.ts), and the BGE-M3 profiles use only its dense output, not its multi-vector mode.
Related ideas
Sources
- Khattab & Zaharia (2020): ColBERT, efficient and effective passage search via contextualized late interaction over BERT
- Santhanam et al. (2022): ColBERTv2, effective and efficient retrieval via lightweight late interaction
- Faysse et al. (2024): ColPali, efficient document retrieval with vision language models