Skip to content

LearnRetrieve

Vocabulary mismatch

The vocabulary problem · Lexical gap

The common case where the person searching and the author of the answer use different words for the same thing, so keyword matching finds nothing.

Background conceptIllustrated explainer

On this page
The question and the answer page share no content wordsA person asks "When can the private jet leave for London?" The query's content words are private, jet, leave and London. The answer page, travel-desk-guide page 3, reads "Charter: once the assistant books, the aircraft departs for FBO within three hours." Its matching ideas are charter, aircraft, departs and FBO, the Farnborough airport code. Exact word matches: 0 of 4, so keyword search scores the page zero. Dashed lines show each pair means the same thing, which vector search can detect.When can the private jet leave for London?travel-desk-guide.pdf · p.3Charter: once theassistant books, theaircraft departs forFBO within threehours.QUERY WORDSPAGE WORDS≠privatecharter≠jetaircraft≠leavedeparts≠LondonFBOShared words: 0 of 4keyword search scores it 0Same meaning: 4 of 4vector search can find it
The answer page shares none of the question’s content words. “Private jet leave London” and “charter aircraft departs FBO” describe the same trip, but an inverted index only connects identical words, so keyword search gives the page nothing. The dashed lines are what a person, or an embedding model, sees: the same meaning in different words.

The problem: the answer is there, and search can’t see it

A new executive assistant asks “When can the private jet leave for London?” The answer is on page 3 of travel-desk-guide.pdf:

Charter: once the assistant books, the aircraft departs for FBO within three hours.

A person reads that and knows it answers the question. Keyword search does not. Strip the stop words (“when”, “can”, “the”, “for”) and the query has four content words: private, jet, leave, London. The page contains none of them. Even with stemming, which reduces “departs” to “depart” and “hours” to “hour”, there is nothing to match. The page scores zero and never appears in the results.

This is vocabulary mismatch: the searcher and the author used different words for the same thing. It is not a rare edge case. In a classic study, Furnas and colleagues asked people to name the same objects and commands, and found that two people chose the same main word less than 20% of the time (The vocabulary problem in human-system communication, 1987). Authors of guides, contracts and manuals write in their own vocabulary, and readers search in theirs.

Why keyword search is blind to it

Lexical search builds an inverted index: for every word, the list of passages that contain it. A query is answered by looking up its words. If a word isn’t in the passage, the index has no entry connecting them, so no ranking formula, BM25 included, can give that passage any credit for it.

Count it for the example:

Query word In the page? Closest word the page uses
private no charter
jet no aircraft
leave no departs
London no FBO (Farnborough)

Zero of four. The page is a perfect answer and a perfect miss.

Normalization helps only at the edges. Stemming maps “departing” to the same stem as “departs”, and Unicode normalization handles accents and ligatures. Neither turns “jet” into “aircraft”.

The fixes, and what each one costs

There are two broad strategies: change the query so it uses the document’s words, or compare meanings instead of words.

Add the missing words to the query.

  • Synonym expansion uses a curated list: if your team’s glossary says private = charter and jet = aircraft, the query becomes (private OR charter) (jet OR aircraft) …. Precise and cheap, but someone has to maintain the list, and it only covers the pairs you thought of.
  • Query rewriting asks a language model to restate the question. It might produce “charter aircraft departure time to FBO”, which now shares words with the page. It costs a model call and can drift from what was asked.
  • Step-back prompting searches a more general version of the question, such as “private aviation booking policy”, alongside the original.

Compare meanings instead of words.

  • Vector search embeds the question and every passage, then ranks by direction. “Private jet leaves for London” and “charter aircraft departs for FBO” land near each other without sharing a single token. This is the main reason vector search exists.
  • HyDE goes one step further: a model drafts a fake answer, and the draft, which is written like a document rather than a question, is what gets embedded.

Use both. Hybrid search runs keyword and vector search together and merges the results, so a query with exact identifiers and descriptive words is covered on both sides.

Where the fixes break

  • Expansion adds noise. Every synonym is another way to match the wrong passage. “Charter” also matches the board’s committee charter.
  • Embeddings blur exact terms. The same fuzziness that bridges “jet” and “aircraft” makes tail number N750MC look like N705MC. Fixing mismatch with vectors alone trades one problem for another.
  • Jargon the model never saw. An embedding model trained on public text doesn’t know that your team calls the Larkspur Studio acquisition “Project Bravo”. Internal names are vocabulary mismatch that only a glossary can fix.
  • Mismatch in the other direction. Sometimes the query has the right words and the page doesn’t: a scanned page with bad OCR or a table extracted without its headers. No query trick fixes text that never made it into the index.

How to spot it

When an answer exists but search misses it, put the query and the answer side by side and count shared content words. If the count is zero or one, you have vocabulary mismatch, and adding a vector branch, a glossary or a rewrite is the fix. If the words overlap and the page still ranks low, the problem is elsewhere: chunking, ranking or the number of candidates you keep.

In DocSlurp

Background concept

Hybrid search adds a vector branch so meaning can match without shared words (core/server/src/search/retrieval-adapters.ts). Synonyms, HyDE, query rewriting and step-back prompting are optional fixes (core/server/src/search/synonym.ts, core/server/src/search/hyde.ts).

Sources

Jump to