Skip to content

LearnIndex & store

Semantic chunking

Placing chunk boundaries where the topic changes, usually detected as a drop in embedding similarity between neighboring sentences, instead of at a fixed size.

Related techniqueGlossary entry

On this page

Semantic chunking places boundaries where the subject changes instead of every N characters. The usual recipe: split the text into sentences, embed each one, compute the cosine similarity between neighboring sentences, and cut where similarity drops sharply, for example below a percentile of all the gaps.

In a board update that moves from “results” to “London expansion” to “Larkspur acquisition”, the sentences within each part resemble each other, and the transitions show up as dips. A fixed-size chunker would cut wherever the character count runs out; a semantic chunker cuts at the dips, so each chunk is about one thing.

The gotchas

  • It costs an embedding per sentence at ingest, on top of embedding the final chunks.
  • Chunk sizes vary widely. A long section on one topic becomes a huge chunk unless a maximum size is also enforced, and a run of short, varied sentences becomes many tiny ones.
  • Topic is not reference. “For trips over $40k, it also needs the CFO’s pre-approval” is on the same topic as the sentence before it, so a semantic chunker probably keeps them together, but nothing guarantees it. When it cuts there, “it” still points nowhere.
  • Gains are corpus-dependent. Well-structured documents with headings often chunk just as well by structure, which is cheaper. Compare against a structural baseline with recall@k before adopting it.

Related approaches avoid boundaries altogether: proposition indexing rewrites text into standalone statements, and chunk overlap duplicates text across cuts.

In DocSlurp

Related technique

DocSlurp does not split on embedding similarity. Its chunker is structural: headings, blocks, sentences, tables, and slide, sheet and media boundaries under an 1,800-character budget (core/worker/src/lib/canonical-document.ts).

Sources

Jump to