Skip to content

LearnIngest & extract

Format detection

File type detection · MIME sniffing

Deciding what kind of file an upload really is, from its declared MIME type, its extension and the magic bytes at the start of the file.

In DocSlurpGlossary entry

On this page

Format detection decides what kind of file an upload really is, so it can be sent to the right extractor. There are three sources of evidence, in increasing order of trust:

  1. The declared MIME type, sent by the browser or client: application/pdf.
  2. The extension: .pdf.
  3. The magic bytes: the first few bytes of the file, which most formats fix by specification.
Starts with Format
%PDF- PDF
PK (50 4B 03 04) zip, which includes .docx, .xlsx, .pptx and .epub
89 PNG PNG image
FF D8 FF JPEG image
RIFF … WAVE WAV audio

The first two are claims made by whoever uploaded the file. The third is evidence. When they disagree, believe the bytes. A file named charter-agreement.pdf that starts with PK is a zip archive, possibly an Office document renamed by accident, possibly something crafted to reach a parser it shouldn’t.

Zip-based formats need one more step: a .docx and a .xlsx have identical magic bytes, so the detector looks inside the archive for word/ or xl/ entries.

The gotcha: a mismatch is a security signal, not just a routing problem. Parsers for complex formats are a classic attack surface, and an attacker who controls the declared type can try to steer a file into the weakest one. Detect from bytes, refuse contradictions rather than guessing, and run every parser in a sandbox. See sandboxing and Office file formats.

In DocSlurp

In DocSlurp

Two steps. The quarantine stage sniffs magic bytes (%PDF-, PK, PNG, JPEG, RIFF and more) and quarantines a file whose declared type contradicts its bytes (core/worker/src/lib/quarantine.ts). Then format-detect maps the MIME type, falling back to the extension, onto one of 16 normalized formats (including unknown) that choose the extractor (core/worker/src/stages/format-detect.ts).

Sources

Jump to