Lexical search
Keyword search · Sparse retrieval
Retrieval that matches the words of the query against the words of each passage, after normalizing both, and ranks by how well they overlap.
In DocSlurpGlossary entry
On this page
Lexical search finds passages that contain the query’s words. Both the query and every passage go through the same normalization: lowercasing, splitting into words, dropping stop words and reducing words to a stem, so “promoted” and “promoting” both become promot. An inverted index maps each stem to the passages that contain it, which makes lookup fast even across millions of passages. A scoring function, usually BM25, then ranks the matches: rare words count more than common ones, and repeated words count with diminishing returns.
For the query “chartering TEB_FBO”, Postgres turns the text into 'charter' & 'teb' <-> 'fbo': the stem of “chartering”, and the route code as two tokens that must appear side by side. A travel-desk page containing both scores; a page about private jets that never says “charter” does not.
The gotcha: all or any
A lexical query must decide whether every word is required. Requiring all of them is precise but brittle: one extra word in a long, chatty question (“please”, “exactly”) and nothing matches. Accepting any word finds something for every query, but a passage can match on one incidental word. Systems usually try the strict version first and back off. DocSlurp’s keyword branch does this, then counts a backed-off match as evidence only when it contains a majority of the query’s content words.
Lexical search can’t match a word the passage doesn’t use, which is vocabulary mismatch. That is why it is usually paired with vector search in hybrid search.
In DocSlurp
In DocSlurp
The keyword branch first requires every query term with Postgres websearch_to_tsquery, then backs off to any term plus three-word phrase windows (core/server/src/services/search.ts). Tables with a LanceDB full-text index use native BM25 instead, unless the query has quotes or a minus sign.