Result deduplication
Near-duplicate removal
Removing results that repeat the same content, from exact copies of a chunk to the same paragraph pasted into two documents.
In DocSlurpGlossary entry
On this page
The same content often comes back more than once. The same chunk can arrive from both keyword and vector search. A file can be uploaded twice. A paragraph can be pasted from q3-earnings-memo.pdf into board-update-2026.pdf. Each repeat takes a slot that a different passage could have used, and a model reading the same paragraph twice gains nothing.
There are three levels of duplicate, each with its own fix:
| Duplicate | Example | How to catch it |
|---|---|---|
| Same chunk | One chunk returned by two retrievers | Compare chunk IDs |
| Same file | growth-plan-2027.pdf uploaded twice |
Hash the file bytes, such as SHA-256 |
| Same text, different source | A paragraph copied into another document | Near-duplicate detection: MinHash or SimHash fingerprints, or embedding similarity between results |
The first two are exact and cheap. The third is the one that hurts, because the copies have different IDs, different filenames and slightly different surrounding text. Maximal marginal relevance handles it at ranking time by penalizing candidates similar to ones already picked.
The gotcha
Not every repeat is redundant. Two versions of a policy that differ by one number are near-duplicates, and the difference may be the answer. Two independent sources stating the same fact are corroboration, which matters for citations. Deduplicate copies, not agreement.
In DocSlurp, rank fusion keeps one copy per chunk ID (core/server/src/search/rrf.ts). An identical re-upload in the same organization reuses the earlier document’s chunks and vectors, matched by source SHA-256 (core/server/src/services/run-creation.ts). A project can also return one result per file with dedupeByFilename (core/server/src/services/search-controls.ts). Near-duplicate text across different documents is not removed.
In DocSlurp
In DocSlurp
Fusion keeps one copy per chunk ID (core/server/src/search/rrf.ts), an identical re-upload in the same organization reuses the earlier document’s chunks and vectors, matched by source SHA-256 (core/server/src/services/run-creation.ts), and a project can keep one result per file with dedupeByFilename (core/server/src/services/search-controls.ts). Near-duplicate text across different documents is not removed.