Skip to content

LearnIngest & extract

PDF text layer

Embedded text · Native text

The text a PDF stores alongside its drawing instructions: positioned glyph runs plus font mappings, from which an extractor rebuilds words, spaces and order.

In DocSlurpIllustrated explainer

On this page
What a PDF page shows, what the file stores, and what extractors rebuildThe page shows the line "Profitable in the first year". The file stores five positioned glyph runs, Profitable, in, the, a run starting with the ligature glyph fi stored as character code 1, and year, separated by adjustments of minus 280, which at 11 points is a 3.08 point gap and no space character. The content stream is BT, font F1 at 11 points, move to 72 700, then a TJ array of the runs and adjustments, then ET. Extractors produce different text: with spaces inferred and the ligature normalized, "Profitable in the first year"; with the space threshold above 3.08 points, "Profitableinthefirstyear"; keeping the ligature, "Profitable in the first year", which a search for first misses; with no ToUnicode entry for code 1, "Profitable in the �rst year".What you seeProfitableinthefirstyearThe same line as stored glyph runsProfitableinthefirstyear4 gaps of 3.08 pt, no space charactersfi is one glyph, stored as code \001What the file storesBT/F1 11 Tf % 11 pt font72 700 Td % move here[(Profitable) -280 (in) -280 (the) -280 (\001rst) -280 (year)] TJETTOUNICODE MAP FOR F1 (OPTIONAL)<01> → U+FB01 "fi", or missingWhat extractors rebuildProfitable in the first yeargaps become spaces, fi becomes f + iProfitableinthefirstyearspace threshold set above the 3.08 pt gapsProfitable in the first yearligature kept: a search for "first" missesProfitable in the �rst yearfont has no ToUnicode entry for code 1Gap between runs: 280 / 1000 × 11 pt = 3.08 pt. A space is inserted only if the gap beats the threshold.threshold 1.50 pt → Profitable in the first year
A PDF stores where to paint glyphs, not sentences; spaces and even letters are rebuilt by the extractor. The line has no space characters, only 3.08-point gaps, and its "fi" is one ligature glyph identified through an optional ToUnicode map. Drag the threshold past 3.08 pt and the words run together. The four outputs on the right all come from the same bytes.

The problem: copy, paste, garbage

Open growth-plan-2027.pdf, select “Profitable in the first year”, copy it and paste it into a chat. Depending on the file and the viewer, you get one of these:

Profitable in the first year
Profitableinthefirstyear
Profitable in the first year
Profitable in the �rst year

The page looks identical every time. What differs is how much of the text the PDF actually stores, and how well the extractor guesses the rest.

What a PDF really stores

A PDF page is a program for painting. Its content stream says which font to use, where to move, and which glyphs to paint. The line above might be stored like this:

BT
  /F1 11 Tf                 % font F1 at 11 points
  72 700 Td                 % move to x = 72, y = 700
  [(Profitable) -280 (in) -280 (the) -280 (\001rst) -280 (year)] TJ
ET

Three things are missing that you might expect a text format to have.

There may be no spaces. The numbers between the strings are horizontal adjustments, in thousandths of the font size. Negative numbers move right. At 11 points, -280 moves 280 / 1000 × 11 = 3.08 points to the right, about the width of a space in Helvetica (278 units, or 3.06 points). The file never says “space”. An extractor sees a 3.08-point gap and must decide whether it is a space, a kerning nudge or a column gap. Guess too high and words run together; guess too low and “Profit able” splits.

Glyphs are codes, not letters. \001 is a character code in font F1. Here it paints the “fi” ligature, a single glyph for the letters f and i. To know which letters a code stands for, the extractor reads the font’s ToUnicode map, an optional table defined in the PDF specification. If the map says code 1 is U+FB01, you get the ligature “fi”, which looks right but won’t match a search for “first” until Unicode normalization rewrites it to “fi”. If the map is missing, the extractor guesses from the glyph name or gives up and emits U+FFFD, the replacement character “�”. Subset fonts with custom encodings can turn a whole page into consistent nonsense.

Order is drawing order. The stream paints in whatever order the authoring tool chose. The footer may be drawn first and the title last. Extractors sort by position to recover reading order, with all the ambiguity that brings.

So an extractor is reconstructing text from evidence: glyph codes, font maps and coordinates. When the evidence is good, the result is exact and essentially free. When it is not, the result can be wrong in ways that look almost right.

Invisible text over scans

Scanned PDFs often carry a text layer too. Some earlier OCR run placed invisible text (text render mode 3: neither filled nor stroked) exactly over the words in the image, so you can select and search it. The page looks like a scan and behaves like text.

That layer is only as good as the OCR that made it, possibly years ago with an old engine. Trusting it means inheriting its mistakes without seeing them. One check: OCR a band of the page again and compare. DocSlurp’s optional page router does this, running Tesseract on a quarter-page band rendered at 100 DPI and refusing to trust the hidden layer when agreement with the fresh read falls below 0.4.

Deciding whether to trust it

A text layer is the best source of text when it is faithful, and a liability when it is not. Signals that it can be trusted:

  • Glyphs map to real characters. Count codes that came out as U+FFFD or private-use code points. DocSlurp’s page probe, part of the optional page router, calls a layer “suspect” above 3% unmapped glyphs, and the router caps trust at 0.2 above 10%.
  • The words look like words. Plausible token shapes, normal punctuation, a sensible ratio of letters to symbols.
  • Text covers what the page shows. A page that is mostly image with ten words of text is a scan with a page number, not a text page.

When the layer fails these checks, throw it away: rasterize the page and read the pixels with OCR or a vision-language model. Choosing per page is page routing.

Where it breaks

  • Math. Formulas are built from glyphs in symbol fonts, positioned above and below the baseline. The text layer holds the symbols but not the structure. See math extraction.
  • Text drawn as shapes. Some tools convert text to vector outlines. The page shows words; the file has no text at all.
  • Right-to-left and vertical scripts. Glyphs may be stored in visual order, so naive extraction reverses Arabic or Hebrew words.
  • Hyphenation and line breaks. “fail-” at the end of one line and “over” at the start of the next are two strings. Joining them correctly is another guess.
  • Hidden and overlapping text. White text on white, text under images, and duplicated layers all extract as if they were visible.

In DocSlurp

In DocSlurp

Most PDF pages are read from their text layer by the sandboxed VM extractor, the native PDF engine or poppler (core/worker/src/extractors/registry.ts). The page probe flags unmapped glyphs (U+FFFD and private-use code points), and the optional page router distrusts a layer when over 10% of glyphs are unmapped (core/worker/src/routing/probe/features.ts, core/worker/src/routing/planner.ts).

Sources

Jump to