Skip to content

LearnIndex & store

Chunk overlap

Repeating some text from the end of one chunk at the start of the next, so a sentence near a boundary appears whole in at least one chunk.

In DocSlurpGlossary entry

On this page

Chunk overlap repeats a stretch of text across a chunk boundary. With 1,000-character chunks and 200 characters of overlap, chunk 2 starts with the last 200 characters of chunk 1. A sentence that would have been cut in half now appears whole in at least one chunk, as long as it is shorter than the overlap.

Take the rule on page 12 of travel-desk-guide.pdf: “Once the assistant sends the request, the Learjet is wheels-up for Farnborough within three hours flat.” A fixed-size cut can land after “wheels-up for Farnborough”, leaving the timing in the next chunk (see the figure on chunking). With enough overlap, the whole sentence also appears at the start of the next chunk, and a search for the turnaround rule can find it intact.

The cost

Overlap is paid for in duplicates. Every overlapped character is embedded, stored and indexed twice, so 20% overlap means roughly 20% more chunks to embed and store. Worse, neighboring chunks now share text, so a query that matches the shared sentence often returns both of them, pushing other evidence out of the top results. Pair overlap with result deduplication, or use sentence-window retrieval, which adds context around a hit at query time instead of duplicating it at index time.

Overlap also can’t fix references that reach further back than the overlap: “From there” three sentences later still loses its antecedent.

DocSlurp’s version is unusual: it leaves base chunks untouched and, when chunkOverlap is on, inserts a separate bridge chunk between each pair, built from the second half of one and the first half of the next.

In DocSlurp

In DocSlurp

Off by default (chunkOverlap in the pipeline settings). When on, DocSlurp keeps its base chunks unchanged and inserts an extra bridge chunk between each neighboring pair, made of the last half of one and the first half of the next; tables are never bridged (core/worker/src/lib/canonical-document.ts).

Sources

Jump to