Skip to content

LearnRank & assemble

Result deduplication

Near-duplicate removal

Removing results that repeat the same content, from exact copies of a chunk to the same paragraph pasted into two documents.

In DocSlurpGlossary entry

On this page

The same content often comes back more than once. The same chunk can arrive from both keyword and vector search. A file can be uploaded twice. A paragraph can be pasted from q3-earnings-memo.pdf into board-update-2026.pdf. Each repeat takes a slot that a different passage could have used, and a model reading the same paragraph twice gains nothing.

There are three levels of duplicate, each with its own fix:

Duplicate Example How to catch it
Same chunk One chunk returned by two retrievers Compare chunk IDs
Same file growth-plan-2027.pdf uploaded twice Hash the file bytes, such as SHA-256
Same text, different source A paragraph copied into another document Near-duplicate detection: MinHash or SimHash fingerprints, or embedding similarity between results

The first two are exact and cheap. The third is the one that hurts, because the copies have different IDs, different filenames and slightly different surrounding text. Maximal marginal relevance handles it at ranking time by penalizing candidates similar to ones already picked.

The gotcha

Not every repeat is redundant. Two versions of a policy that differ by one number are near-duplicates, and the difference may be the answer. Two independent sources stating the same fact are corroboration, which matters for citations. Deduplicate copies, not agreement.

In DocSlurp, rank fusion keeps one copy per chunk ID (core/server/src/search/rrf.ts). An identical re-upload in the same organization reuses the earlier document’s chunks and vectors, matched by source SHA-256 (core/server/src/services/run-creation.ts). A project can also return one result per file with dedupeByFilename (core/server/src/services/search-controls.ts). Near-duplicate text across different documents is not removed.

In DocSlurp

In DocSlurp

Fusion keeps one copy per chunk ID (core/server/src/search/rrf.ts), an identical re-upload in the same organization reuses the earlier document’s chunks and vectors, matched by source SHA-256 (core/server/src/services/run-creation.ts), and a project can keep one result per file with dedupeByFilename (core/server/src/services/search-controls.ts). Near-duplicate text across different documents is not removed.

Sources

Jump to