Skip to content

LearnIndex & store

Chunking

Text splitting

Splitting extracted text into passages small enough to embed, index and hand to a model, each one becoming a unit that search can find and cite.

In DocSlurpIllustrated explainer

On this page
travel-desk-guide.pdf p.12the turnaround rule

Meridian's travel desk books the Learjet 75 (N750MC) at TEB. Once the assistant sends the request, the Learjet is wheels-up for Farnborough within three hours flat. From there, the flight to London takes 7h 10m. Catering and a ground car are arranged with the booking, and the CFO pre-approves trips above $40k.

3 chunks. The rule is split across chunks 1 and 2: one says the Learjet is wheels-up, another holds the timing with no subject.

A chunk boundary decides which facts can be found together. At 140 characters, a fixed-size cut falls after "wheels-up for Farnborough", so chunk 2 opens with "within three hours flat" and never says what is that fast. Switch to sentence-aware and the rule survives whole, but chunk 3 now starts with "From there", cut off from the sentence it refers to. Drag the size to see where each rule breaks.

The problem: an answer split in two

Page 12 of travel-desk-guide.pdf says how fast the jet goes: once the assistant sends the request, the Learjet is wheels-up within three hours flat. You ask “How fast can my assistant get the Learjet to London?” and search returns a passage that says the Learjet “is wheels-up for Farnborough”. The next passage starts with “within three hours flat.” Neither one answers the question, and the model that reads them has to guess.

The document was fine. The text extraction was fine. The cut was in the wrong place.

Why documents get cut at all

Search engines and language models don’t work on whole documents. An embedding model turns a passage into one vector, and a 40-page PDF squeezed into one vector says “this is about executive travel” and nothing more specific. A model’s context window holds a limited amount of text, and stuffing it with whole documents crowds out other evidence. Citations also need something smaller than a file to point at.

So ingestion splits each document into chunks: passages that are embedded, indexed, retrieved and cited as units. Every later step works on chunks, so the chunker sets a hard limit on retrieval quality. Search can rank chunks well or badly, but it can only ever return what one chunk says.

How a chunker decides where to cut

The figure shows two common rules on the same 311-character paragraph.

Fixed-size cuts every N characters (or tokens), no matter what is there. At 140 characters the paragraph becomes three chunks, and the first cut lands after “wheels-up for Farnborough”. The turnaround rule is now split across chunks 1 and 2.

Sentence-aware packs whole sentences into each chunk until the next sentence would exceed N, then starts a new chunk. At 140 characters it produces four chunks, and the rule (one 104-character sentence) stays whole in chunk 2. If a single sentence is longer than N, it still has to be hard-split, so drag the size below 104 and the rule breaks again.

Neither rule understands the text. Sentence-aware splitting protects sentences, not ideas. In the figure, chunk 3 begins “From there, the flight to London takes 7h 10m.” Retrieved alone, “From there” refers to nothing.

Real chunkers stack several rules, from strongest boundary to weakest:

  1. Structure first. Headings, slides, spreadsheet tabs, table edges and speaker turns are natural breaks. A heading should start a chunk, and a table should not be cut through the middle of a row.
  2. Then blocks and sentences. Paragraphs and sentences are packed together until the size budget is reached.
  3. Hard splits last. Only text that is longer than the budget on its own gets cut mid-sentence.

Size is a trade-off, not a setting to max out

Small chunks make precise matches. A 200-character chunk about jet turnaround is only about jet turnaround, so its vector sits close to questions about it. But small chunks cut more often, so more facts get separated from their conditions, subjects and units.

Large chunks keep context together, but each vector now averages several topics, and a question about one sentence has to compete with everything else in the chunk. Large chunks also spend more of the model’s context window per result.

There is no universal best size. It depends on the documents (dense contracts versus chatty transcripts), the embedding model and the questions. Measure it: build a set of questions with known answers, then compare recall@k across chunk sizes on your own corpus.

Fixes for bad cuts

  • Chunk overlap repeats text across a boundary, so a sentence near a cut appears whole somewhere. It costs duplicate storage and duplicate results.
  • Contextual retrieval prepends a short note to each chunk saying which document and section it came from, so “From there” at least carries its heading.
  • Small-to-big retrieval searches small chunks for precision, then hands the model the larger passage around each hit.
  • Semantic chunking moves cuts to where the topic changes, measured with embeddings.
  • Proposition indexing has a model rewrite text into standalone statements, so no fact depends on its neighbor.

Where it breaks

  • Pronouns and references. “It”, “this rule” and “from there” lose their meaning when the sentence they point to is in another chunk. No size or rule fully prevents this.
  • Tables and lists. A fixed-size cut through a table separates values from their column headers. Tables need their own chunking that keeps headers with rows.
  • Tokens versus characters. Embedding models limit input in tokens, but many chunkers count characters. The ratio changes with language and content, so code or non-English text can exceed a token limit that prose never reaches.
  • Re-chunking means re-indexing. Changing the chunker changes every chunk and every vector. Treat chunking settings as part of the index, not a query-time option.

In DocSlurp

DocSlurp chunks each document’s canonical block structure with a budget of 1,800 characters. A heading always starts a new chunk, and so does a change of slide, spreadsheet sheet or media segment. Blocks are packed together up to the budget. A single block longer than 1,800 characters is cut into 1,800-character slices. Tables get their own chunks, and a table too long for one is split by record, each cell labeled with its table, row and column. Every chunk keeps its heading path and page locator, which is what lets a citation open the right page. Overlap is off unless a project turns on chunkOverlap.

In DocSlurp

In DocSlurp

DocSlurp chunks at up to 1,800 characters (not tokens), starting a new chunk at each heading and at slide, sheet and media-segment boundaries, and chunking tables record by record (core/worker/src/lib/canonical-document.ts, core/worker/src/pipeline/chunking.ts). Each chunk keeps its heading path and page locator.

Sources

Jump to