Skip to content

LearnIngest & extract

Document summary

Summary index

A short model-written paragraph describing a whole document, generated once at ingestion and stored, often embedded, so search and chat can treat the file as a unit.

In DocSlurpGlossary entry

On this page

A document summary is a short paragraph, written by a language model at ingestion time, that describes a whole document: its topic, purpose and main entities. It is stored with the document and often embedded and indexed like a chunk.

Chunks are good at matching specific facts and bad at answering questions about whole files. Ask “which document covers booking the company jet?” and the best chunk from travel-desk-guide.pdf might be step 7 of 20, which never says “booking the company jet” at all. A summary such as “Step-by-step guide for executive assistants booking the Learjet 75 through the travel desk, with CFO pre-approval, catering and ground transport” matches the question directly, and points at the right file.

Summaries are useful for:

  • Document-level search and routing a question to the right files before searching their chunks.
  • Display: a one-line description in search results and file pickers.
  • Broad questions in chat, where the answer is about a set of documents rather than one passage.

Long documents don’t fit in one cheap model call, so summarizers either sample representative excerpts or summarize sections and then summarize the summaries. See map-reduce summarization and RAPTOR, which builds a whole tree of them.

The gotcha: a summary is generated text, not source text. It can omit the one detail you needed or state something the document only implied. Use it to find documents, and cite the document’s own passages for any claim. A summary built from sampled excerpts should say so, rather than imply it covered everything.

In DocSlurp

In DocSlurp

Opt-in per project (summarize, off by default). The document-summary stage asks a hosted text model for one paragraph (at most 256 output tokens) from up to 10,000 bytes of extracted text, sampling excerpts from long documents, then stores and embeds it (core/worker/src/stages/document-summary.ts, summary-input.ts). Privacy modes that forbid hosted models skip it.

Sources

Jump to