Skip to content

LearnGraphs & hierarchy

Map-reduce summarization

Hierarchical summarization

Summarizing text too long for one model call by summarizing each piece separately (map), then combining the partial summaries into one (reduce).

Related techniqueGlossary entry

On this page

Map-reduce summarization handles text too long for one model call. Split it into pieces that fit, summarize each piece on its own (the map step, which can run in parallel), then summarize the summaries (the reduce step). If the partial summaries are still too long together, reduce again, building a tree.

Summarize all eight documents of the finance corpus: map produces one paragraph each for growth-plan-2027.pdf, q3-earnings-memo.pdf, board-update-2026.pdf and the rest; reduce merges eight paragraphs into one page. The same pattern answers a question across a corpus: map asks each piece “what does this say about FY2027 earnings?”, reduce combines the answers. Microsoft’s GraphRAG global search does this over community summaries, scoring each partial answer for helpfulness and dropping the useless ones before the reduce.

The gotcha

Information only flows upward. The reduce step sees nothing the map step left out, so a detail one map call judged minor (“EBITDA guidance raised from $9.1M to $10.6M”) can never reach the final summary, even if it matters in context. Contradictions between pieces also get smoothed into vague agreement. Asking map calls to keep numbers, names and disagreements verbatim reduces the loss.

In DocSlurp

DocSlurp does not use map-reduce. A document summary is one model call. For a long document, the input is a sample of at most 10 pages (the first three, the last, and one from each stretch between), clipped to about 10 KB of text and labeled as sampled so the model doesn’t claim to have read everything (core/worker/src/stages/summary-evidence.ts). That trades completeness for one bounded call per document.

In DocSlurp

Related technique

DocSlurp does not use map-reduce. A document summary is one model call; long documents are represented by a labeled sample of at most 10 pages (core/worker/src/stages/summary-evidence.ts). Community summaries are written per community but never reduced into a global answer.

Sources

Jump to