Full-text search
FTS
Searching the words inside documents, not just titles or fields, using an inverted index and a ranking function such as BM25.
In DocSlurpGlossary entry
On this page
Full-text search finds documents by the words inside them and ranks the matches. It is the engine behind most search boxes, and the “keyword” half of hybrid search.
Two steps make it different from scanning text with grep:
- Analysis. Text is split into terms and normalized: lowercased, stripped of stop words, stemmed, sometimes folded from “café” to “cafe”. The same analysis runs on documents at index time and on queries at search time, so “Jet charters” in a chunk matches the query “jet charter”.
- Indexing and ranking. Normalized terms go into an inverted index, so a query reads short posting lists instead of every document. Matches are then ordered by a relevance function, usually BM25.
Most engines also support query operators: required and excluded terms, OR, and quoted phrases, which need word positions stored in the index.
The gotcha
Full-text search matches words, not meaning. A chunk that says “the Learjet charter is booked” never matches “private jet”, and a query with a typo matches nothing unless the engine adds fuzzy matching. It is also only as good as its analysis: an English stemmer on French text, or a tokenizer that splits us-east-1 into three pieces, quietly changes what can be found. These are the gaps vector search fills, which is why the two are usually combined.
DocSlurp runs two full-text engines over chunk text: LanceDB’s BM25 index by default, and PostgreSQL full-text search for phrase and negation queries.
In DocSlurp
In DocSlurp
DocSlurp has two full-text engines: LanceDB’s native BM25 index on chunk text (the default, core/server/src/search/lancedb.ts) and PostgreSQL tsvector search, which handles phrase and negation queries (core/server/src/services/search.ts).