Skip to content

LearnRetrieve

Metadata filtering

Filtered vector search · Pre-filtering and post-filtering

Restricting search to passages whose attributes match a condition, such as a tag, file type, page range or date, alongside the similarity ranking.

In DocSlurpIllustrated explainer

On this page

Query: What are first-year bookings for the London office?  ·  filter tag = uk  ·  top 3

Post-filter search top k, then filter

  1. 1growth-plan-2027.pdfus0.91fetched, then dropped
  2. 2launch-review-uk.pdfuk0.88returned
  3. 3q3-earnings-memo.pdfus0.85fetched, then dropped
  4. 4board-update-2026.pdfus0.83not fetched
  5. 5london-office-plan-uk.pdfuk0.80not fetched
  6. 6investor-call.mp4us0.76not fetched
  7. 7expansion-forecast-uk.xlsxuk0.71not fetched
  8. 8london-office-tour-uk.jpguk0.66not fetched

Returned 1 of 3

Pre-filter filter, then search top k

  1. 1growth-plan-2027.pdfus0.91filtered out first
  2. 2launch-review-uk.pdfuk0.88returned
  3. 3q3-earnings-memo.pdfus0.85filtered out first
  4. 4board-update-2026.pdfus0.83filtered out first
  5. 5london-office-plan-uk.pdfuk0.80returned
  6. 6investor-call.mp4us0.76filtered out first
  7. 7expansion-forecast-uk.xlsxuk0.71returned
  8. 8london-office-tour-uk.jpguk0.66not fetched

Returned 3 of 3

Filter after a top-k search and you get fewer results than you asked for. Eight passages ranked by similarity; four carry the uk tag. Post-filtering fetches the top 3, finds one uk passage among them and returns 1 of 3. Pre-filtering removes the US passages first and returns 3 of 3. Drag k to 1: post-filtering returns nothing at all. Similarities are illustrative.

The problem: “only the UK documents”

Meridian & Co. is opening a London office, and the launch team asks “What are first-year bookings for the London office?” They want the answer from UK documents only; the US plans carry different numbers. Every passage in the index carries metadata: its file, tags, type, page number, upload date. So you add a filter, tag = uk, and ask for the top 3.

There are two obvious ways to combine a filter with a similarity search, and they give different answers.

Post-filtering: search, then filter

Run the vector search as usual, take the top k, then drop anything that fails the filter.

Take the eight passages in the figure, already ranked by similarity to the question. Four are tagged uk, at ranks 2, 5, 7 and 8. Post-filtering with k = 3 fetches ranks 1 to 3, then throws away the two US passages. You asked for 3 results and got 1. At k = 1 you’d get none: the single nearest passage is a US one.

Post-filtering is simple, and it works with any index. It fails exactly when the filter is selective, because the top k of the whole corpus contains few passages that pass. The usual patch is to over-fetch, say 10 × k, and filter that. That costs more, and there is still no guarantee: if uk passages are 2% of the corpus and none is in the top 30, you still return nothing.

Restrict the search to passages that pass the filter, then take the top k among those.

On the same eight passages, pre-filtering at k = 3 considers only the four uk passages and returns ranks 2, 5 and 7. Three results, all relevant to the filter, in similarity order. Drag k in the figure: pre-filtering returns k results until it runs out of uk passages, while post-filtering returns only the uk passages that happen to fall inside its window.

Pre-filtering gives the correct answer by construction, which is why it is the default in most vector databases. The cost moves into the index. Approximate nearest neighbor indexes such as HNSW are graphs built over all vectors; when most nodes are filtered out, the graph walk keeps hitting forbidden nodes and has to search further or switch to exact search over the filtered set. Research systems such as ACORN build the graph so filtered walks stay connected.

A third option: separate indexes

When a filter is fixed and coarse, such as one customer’s data, don’t filter at all: give each value its own index. Separate indexes make cross-tenant leaks structurally harder, which matters for multi-tenancy, and every search is a plain unfiltered search. It only works for a small number of large partitions, not for arbitrary tags or date ranges.

Where it breaks

  • Metadata is only as good as ingestion. A filter on language = de silently excludes every German page whose language was never detected. Metadata extraction quality is filter quality.
  • Passage-level versus document-level. A tag usually belongs to a document, but a page range or a heading belongs to a passage. Systems often apply the two at different stages, which can reintroduce post-filtering for the passage-level part.
  • Filters hidden in the question. “What does page 12 of the growth plan say?” contains a filter, but a search box treats it as text to embed. Self-query retrieval extracts it.
  • Over-filtering looks like no answer. A filter that matches nothing returns an empty list that is easy to mistake for “the corpus doesn’t know”. Report which filters were applied.

When to use which

Use pre-filtering by default, especially for selective filters and anything security-related, where a missing result is a bug. Post-filtering with generous over-fetch is acceptable for loose filters that most passages pass. Use separate indexes for tenant or project boundaries.

In DocSlurp

Each project has its own LanceDB database, so the project boundary is a separate index rather than a filter. Inside it, document-level filters (documentId, tags, mimeTypes, filenameContains, runId) are resolved to a document list first, and the vector search runs with that list as a pre-filter. The code comment explains why: over-fetching a global top-k and filtering afterwards “can miss every scoped document”. Passage-level filters (pageFrom/pageTo, heading, language, date) narrow the search to documents that have at least one matching chunk, then are checked against the returned chunks, so a narrow page range can return fewer hits than limit. The keyword branch applies all filters inside its SQL query.

In DocSlurp

In DocSlurp

Document filters (documentId, tags, mimeTypes, filenameContains, runId) narrow vector search before nearest-neighbor selection (core/server/src/search/lancedb.ts, core/server/src/search/retrieval-scope.ts). Page, heading, language and date filters also narrow to matching documents first, but are checked against the returned chunks afterwards, so they can return fewer hits.

Sources

Jump to