Skip to content

LearnIngest & extract

Ingestion pipeline

Document pipeline

The ordered stages that turn an uploaded file into searchable records: check it, detect its format, extract text, enrich it, chunk it, embed it and index it.

In DocSlurpGlossary entry

On this page

An ingestion pipeline is the ordered sequence of stages that turns an uploaded file into searchable, citable records. Every search and RAG system has one, even if it is three lines of glue code. A fuller one looks like this:

  1. Intake: check the file is safe and what it claims to be (format detection, size limits, malware and zip-bomb checks).
  2. Extract: turn it into text with structure and locations: the PDF text layer, OCR, a vision-language model or speech-to-text, ending in Markdown conversion.
  3. Enrich (optional): PII detection, document summaries, structured fields, entities for a knowledge graph.
  4. Index: chunking, embedding, and writing to keyword and vector indexes.

Each stage consumes the previous one’s output, so errors flow downhill. If page 12 of growth-plan-2027.pdf was extracted with its columns interleaved, the chunks are spliced, the embeddings are muddled, and no reranker can recover the sentence. When an answer is missing, check the extracted text before tuning retrieval.

Good pipelines also share some operational habits:

  • Checkpoints. Record each stage’s output per document (and per page for long files), so a crash or retry resumes instead of starting over. See idempotency.
  • Per-stage metrics. Pages per minute and cost per stage show where time and money go.
  • Isolation. Parsers read hostile bytes; run them in a sandbox away from secrets. See sandboxing.

The gotcha: a stage that “succeeds” with empty or garbled output is worse than one that fails. Check that extracted text is plausible before indexing it.

In DocSlurp

In DocSlurp

Runs follow a fixed stage order in four phases (intake, prep, extract, index), each stage switched on by project features: quarantine, format-detect, to-canonical, extraction, optional vision, PII, summary and structured fields, then embed and index (core/packages/shared/src/pipeline.ts, core/worker/src/stages/index.ts). Stages checkpoint per document and page, so a retry resumes.

Sources

Jump to