Stemming and lemmatization
Lemmatization
Reducing words to a shared base form before indexing, so "books", "booked" and "booking" all match the query "book".
In DocSlurpGlossary entry
On this page
Stemming and lemmatization reduce words to a shared base form before indexing and querying, so different forms of a word match each other.
A stemmer applies suffix-stripping rules. The Porter stemmer (1980) and its successor, the Snowball English stemmer, turn “books”, “booked” and “booking” into “book”, and “flights” into “flight”. The result need not be a real word: “company” becomes “compani”. What matters is that the document and the query are reduced the same way, so a query for “booked charter flights” finds a chunk that says “books charter flights”.
A lemmatizer uses a vocabulary and the word’s grammatical role to return the dictionary form, the lemma: “ran” becomes “run” and “better” can become “good”. It is more accurate and slower, and needs language-specific resources.
The gotchas
- Over-stemming merges unrelated words. Porter reduces “university” and “universe” to the same stem, so one matches the other.
- Under-stemming misses related words. Irregular forms such as “ran” and “run” stay apart under a stemmer.
- It is language-specific. An English stemmer applied to German or French text produces nonsense stems, and mixed-language corpora need per-document handling.
- Names and codes suffer. A stemmer can mangle product names or identifiers that happen to end in “-ing” or “-s”. Exact identifiers are often better matched by phrase search or a separate unstemmed field.
Vector search does not need stemming: embeddings already place “booked” and “booking” close together.
In DocSlurp
In DocSlurp
Both keyword engines stem English: PostgreSQL through the english text search configuration (core/server/src/services/search.ts) and LanceDB through its FTS index (stem: true, language: "English", core/worker/src/stages/lancedb-maintenance.ts). Neither lemmatizes.