Skip to content

LearnIndex & store

TF-IDF

Term frequency–inverse document frequency

Weights a word in a document by how often it appears there (term frequency) times how rare it is across the collection (inverse document frequency).

Background conceptGlossary entry

On this page

TF-IDF scores how important a word is to one document within a collection. It multiplies two numbers:

  • Term frequency (tf): how often the word appears in this document.
  • Inverse document frequency (idf): how rare the word is across all documents, commonly ln(N / n) for N documents of which n contain the word.

Using the four chunks from the inverted index example, “expansion” appears in 1 of 4 chunks, so idf = ln(4 / 1) = 1.386. “plan” appears in 3 of 4, so idf = ln(4 / 3) = 0.288. A word in all four chunks would get ln(4 / 4) = 0: it can’t tell chunks apart, however often it appears. In chunk 4, which mentions each word once, “expansion” carries almost five times the weight of “plan”.

Karen Spärck Jones introduced the idf idea in 1972, and tf-idf vectors were the standard way to represent documents for decades. Each document becomes a sparse vector with one weight per vocabulary word, compared with cosine similarity.

The gotcha

Raw tf grows without limit, so a document that repeats a word fifty times wins, and long documents win by having room to repeat. BM25 is tf-idf with two fixes: term frequency saturates, and scores are normalized by document length. When people say “keyword search” today they usually mean BM25, not plain tf-idf.

In DocSlurp

Background concept

DocSlurp does not rank with plain TF-IDF. Its BM25 paths use the same IDF idea; the local reranker computes IDF over the candidate pool as ln(1 + (N − n + 0.5) / (n + 0.5)) (core/server/src/search/rerank.ts).

Sources

Jump to