Skip to content

LearnIndex & store

BM25

Okapi BM25 · Best Matching 25

A keyword ranking formula that scores a passage by how often it contains each query term, how rare that term is, and how long the passage is.

In DocSlurpIllustrated explainer

On this page
BM25 saturates repeated terms and discounts long chunksLeft chart: the term-frequency contribution in BM25 for a chunk of average length, plotted for 0 to 10 mentions of a term. A raw count rises in a straight line. With k1 = 1.2 the contribution is 1 for one mention, 1.57 for three and 1.96 for ten, approaching a ceiling of 2.2. With k1 = 0.5 it flattens almost at once, reaching 1.43 at ten mentions; with k1 = 3 it reaches 3.08 at ten. Right chart: three mentions of a term in chunks half, equal to, twice and four times the average length, with k1 = 1.2 and b = 0.75, contribute 1.76, 1.57, 1.29 and 0.96. With b = 0 all four would contribute 1.57.Repeats saturatecontribution of one term, average-length chunk0123450246810times the term appears (tf)raw countk1=0.5k1=3k1=1.2ceiling k1 + 1 = 2.2Long chunks are discountedtf = 3 in every chunk, k1 = 1.2, b = 0.7501231.760.5×1.571×1.292×0.964×chunk length ÷ average length
The tenth mention of "EBITDA" is worth far less than the first. With k1 = 1.2, one mention scores 1, three score 1.57 and ten only 1.96, never passing 2.2. On the right, the same three mentions count for less in a long chunk, because they are a smaller share of what it says. Set b to 0 and all four bars match; raise k1 and repetition matters more.

The problem: counting matches is not enough

You search the finance docs for “EBITDA guidance”. An inverted index finds every chunk containing either word in milliseconds. Now you have 300 matches and need an order.

Counting matches fails quickly. A 3,000-word appendix that repeats “guidance” forty times beats a two-sentence chunk that states the new EBITDA guidance. A chunk containing “guidance” (a word in half the corpus) ranks the same as one containing “EBITDA” (a word in three chunks). You need a score that knows which words are rare, stops rewarding repetition, and doesn’t favor long text just for being long.

Three ideas in one formula

BM25 (“Best Matching 25”, from the Okapi system of the 1990s) adds up a score for each query term found in a chunk:

score(chunk) = Σ over query terms   IDF(term) × tf × (k1 + 1) / (tf + k1 × (1 − b + b × len / avg_len))

It looks dense, but it is three separate ideas.

1. Rare words count more (IDF). Inverse document frequency is high for words in few chunks and near zero for words in almost all of them. With N chunks and n containing the term, one common form is ln(1 + (N − n + 0.5) / (n + 0.5)). Matching “EBITDA” tells you a lot; matching “the” tells you nothing. This part is inherited from TF-IDF.

2. Repetition saturates (k1). For a chunk of average length, the term-frequency part is tf × (k1 + 1) / (tf + k1). One mention scores 1. With the common default k1 = 1.2, three mentions score 1.57 and ten score 1.96, and no number of mentions passes 2.2 (that is, k1 + 1). The left side of the figure plots this curve. Small k1 means “mentioned at all” is nearly everything; large k1 makes the score closer to a raw count.

3. Long chunks are discounted (b). The term 1 − b + b × len / avg_len grows with length, which shrinks the score. Three mentions in a chunk half the average length are a big share of what it says; three mentions in a chunk four times the average are a passing reference. With b = 0.75, the figure’s right side shows the same three mentions scoring 1.76, 1.57, 1.29 and 0.96 as length grows. With b = 0 length is ignored; with b = 1 the score is fully scaled by length.

A worked example

Four chunks, average length 20 words, query “EBITDA guidance”, k1 = 1.2, b = 0.75:

Chunk Length “EBITDA” “guidance”
A 10 1 1
B 40 3 2
C 20 0 1
D 10 0 0

“EBITDA” is in 2 of 4 chunks: IDF = ln(1 + 2.5 / 2.5) = ln 2 = 0.693. “guidance” is in 3 of 4: IDF = ln(1 + 1.5 / 3.5) = 0.357.

Chunk A is half the average length, so the length factor is 1 − 0.75 + 0.75 × 0.5 = 0.625, and k1 × 0.625 = 0.75. Each term with tf = 1 scores 1 × 2.2 / (1 + 0.75) = 1.257. Total: 0.693 × 1.257 + 0.357 × 1.257 = 1.320.

Chunk B is twice the average, so the factor is 0.25 + 1.5 = 1.75, and k1 × 1.75 = 2.1. “EBITDA” with tf = 3 scores 6.6 / 5.1 = 1.294; “guidance” with tf = 2 scores 4.4 / 4.1 = 1.073. Total: 0.693 × 1.294 + 0.357 × 1.073 = 1.280.

Chunk C scores 0.357 × 1.0 = 0.357, and D scores 0.

A wins with one mention of each word. B has five matches to A’s two, but it is four times longer. Set b = 0 and B wins, 1.580 to 1.050: without length normalization, longer chunks win by having room to repeat words.

Where it breaks

  • It only sees exact terms. BM25 scores zero for a chunk that says “raised the profit outlook” instead of “EBITDA guidance”. That is vocabulary mismatch, the reason to pair BM25 with vector search in hybrid search.
  • Scores are not comparable across queries or corpora. IDF depends on the collection, and the sum grows with the number of query terms. A BM25 score of 14 is not “better” than 9 from a different query. This is why fusion with vector results uses ranks (reciprocal rank fusion) or explicit score normalization.
  • Word order is invisible. “Guidance EBITDA” and “EBITDA guidance” score the same, and so does a chunk where the two words are paragraphs apart. Phrase queries need positions in the index.
  • IDF is fragile on small or skewed sets. In a tiny collection, or one where every document is about EBITDA, IDF stops separating useful words from common ones.
  • The defaults are defaults. k1 between 1.2 and 2.0 and b = 0.75 work well for prose. Short, uniform passages such as titles or product names often do better with less length normalization. Tune on your own queries if you tune at all.

When to use it

BM25 is the default ranking for keyword search, and a strong baseline that is still hard to beat on exact-term queries: error codes, names, part numbers, legal clause titles. It needs no training and no GPU. Use it alone when users type the words that appear in your documents, and alongside vector search when they don’t.

In DocSlurp

Keyword search queries LanceDB’s native BM25 index over chunk text by default, filtered to the documents in scope before ranking. Queries with quoted phrases or -negation go to PostgreSQL instead, because the BM25 index stores no positions; PostgreSQL ranks with ts_rank_cd, a different formula (see PostgreSQL full-text search). The optional local reranker is its own small BM25 (k1 = 1.5, b = 0.75) that computes IDF over just the candidate passages and adds a bonus when query terms appear next to each other. It reads words, not meaning, despite running in the reranking step.

In DocSlurp

In DocSlurp

Keyword search uses LanceDB’s native BM25 full-text index by default (core/server/src/search/lancedb.ts), falling back to PostgreSQL ts_rank_cd, which is not BM25, for quoted phrases, negation and tables without the index (core/server/src/services/search.ts). The local reranker is a separate BM25 with k1 = 1.5, b = 0.75 (core/server/src/search/rerank.ts).

Sources

Jump to