Stop words
Very common words such as "the", "of" and "is" that a keyword index drops because they appear almost everywhere and say little about relevance.
In DocSlurpGlossary entry
On this page
Stop words are very common words, such as “the”, “of”, “is”, “when” and “does”, that a keyword index leaves out. They appear in nearly every document, so they carry almost no information about which document is relevant, and their posting lists would be enormous.
With stop words removed, the question “When does the London office take bookings?” is searched as “London office take bookings”. The index is smaller, queries read shorter lists, and ranking focuses on the words that distinguish documents. BM25 already gives such words almost no weight through IDF, so removal mostly saves space and time.
The gotcha
Some queries are made of stop words, or depend on them. “To be or not to be” and the band “The Who” disappear entirely. “Flights to London” and “flights from London” become the same query. A query consisting only of stop words can produce an empty search, which engines must handle explicitly instead of returning an error.
Lists also differ: the English list in one engine is not the list in another, and a word like “will” or “may” can be a stop word in one and meaningful (a name, a month) in another. Many modern engines keep stop words in the index and let IDF down-weight them, so phrase queries still work.
In DocSlurp, both keyword indexes drop English stop words, and the local BM25 reranker uses its own short list of 32 words.
In DocSlurp
In DocSlurp
LanceDB’s FTS index removes English stop words (core/worker/src/stages/lancedb-maintenance.ts), PostgreSQL’s english configuration does too, and the local BM25 reranker uses its own 32-word list (core/server/src/search/rerank.ts). When websearch_to_tsquery fails on a degenerate query, search retries with plainto_tsquery (core/server/src/services/search.ts).