Skip to content

LearnIngest & extract

Metadata extraction

Pulling facts about a document, such as title, author, dates, page or slide numbers and timestamps, out of the file or its content and storing them as fields.

In DocSlurpGlossary entry

On this page

Metadata extraction pulls facts about a document out of the file and its content and stores them as fields: title, author, created and modified dates, language, page count, and the location of every passage (page, slide, sheet range or timestamp).

There are two sources:

  • Embedded metadata. PDFs have an info dictionary and XMP metadata; Office files have document properties; web pages have <title> and Open Graph tags such as og:title; media files have container tags.
  • Content. The first large heading, a date line under the title, the “Prepared by” field on a cover page.

Embedded metadata is cheap and often wrong. PDF titles of “Microsoft Word - Document1.docx”, “Untitled” or “Slide 1” are routine, because the field was filled in by software, not by a person. Treat it as a hint: reject values that are file names, tool names or generic words, then fall back to the content. For growth-plan-2027.pdf, a heading like “Growth Plan FY2027” on page 1 beats a metadata title of “draft_final_2.docx”.

Why it matters: metadata becomes filters (“only documents from 2024”), display names in search results, and evidence in citations. Location metadata is the most valuable of all, because it lets every chunk point back to “page 12” or “03:10” in the source. See metadata filtering and provenance.

The gotcha: metadata is untrusted input, written by whoever made the file. A title can contain control characters, be 10,000 characters long, or contain text crafted to look like instructions. Clean it, cap its length, and never place it in a model prompt as if it were trusted.

In DocSlurp

In DocSlurp

Titles come from HTML og:title or <title>, otherwise from an early heading in the first two pages; producer prefixes, filenames and generic titles are rejected (core/packages/shared/src/document-title.ts). Blocks carry page, slide and sheet-range locations (core/worker/src/lib/office-structure.ts), and media blocks carry start and end seconds (core/worker/src/media/canonical.ts).

Sources

Jump to