Vision-language model
VLM · Multimodal LLM
A language model that also takes images as input. For documents, it reads a page image and writes the content out as Markdown, tables and LaTeX in one step.
In DocSlurpIllustrated explainer
On this page
The problem: a page that needs three specialists
Page 7 of launch-review.pdf is a scan. It has a paragraph, a small table of London launch steps, a bar chart of quarterly bookings and one formula for operating margin. The classic pipeline sends it through a chain of specialists: layout analysis to find the regions, OCR for the paragraph, table extraction for the grid, something else for the chart, and a math recognizer for the formula. Every handoff can fail, and the errors compound.
A vision-language model does it in one call. Give it the page image and an instruction such as “transcribe this page as Markdown”, and it writes back the heading, the paragraph, a pipe table, the chart’s values as a table, and the formula in LaTeX.
How a page becomes tokens
A vision-language model (VLM) is a large language model with an image encoder attached.
- Cut the image into patches. The encoder splits the page into small squares, often 14 to 32 pixels on a side, depending on the model.
- Turn each patch into a vector. A vision transformer reads the patches together and produces one vector per patch (models often merge neighbors to save tokens).
- Project into the language model. A small adapter maps those vectors into the same space as the language model’s word embeddings. To the language model, the image is now a long run of extra input tokens.
- Generate text. The language model writes its answer one token at a time, attending to the image tokens and the prompt together.
The numbers explain the cost. A US Letter page rendered at 150 DPI is 8.5 × 150 = 1,275 by 11 × 150 = 1,650 pixels. Cut into 32-pixel patches, that is 40 × 52 = 2,080 patches (rounding up at the edges). A 384 × 384 thumbnail is only 12 × 12 = 144. Patch sizes and resizing rules vary by model, so treat these as illustrative, but the shape holds: image tokens are input tokens, and a sharp page is a long prompt.
The prompt is part of the extractor
A VLM does what it is asked, so the prompt defines the output format. DocSlurp’s page prompt asks for GitHub-flavored Markdown, every visible word verbatim, columns finished before the next one starts, tables as pipe tables, and every formula in LaTeX ($...$ inline, $$...$$ for display). Each clause exists because a model did the opposite once: summarized instead of transcribing, read straight across two columns, turned a list into a table, or wrote math as Unicode symbols.
Where it breaks
- Fluent invention. OCR shows its doubt as garbled characters. A VLM fills a smudged digit with a plausible one, and the result looks exactly like a correct transcription. This is hallucination at the extraction stage, before any question is asked.
- No confidence, often no boxes. Most VLM output is plain text with no per-word confidence and no coordinates. You cannot easily flag the uncertain cell or highlight the cited word on the page.
- Skipping and looping. On dense pages models sometimes skip repetitive rows, stop early at an output limit, or repeat a line until the limit is hit.
- Text in the image is still text. A page that says “ignore previous instructions” is content to transcribe, not an instruction to follow. Prompts and output handling must treat it that way.
- Cost and privacy. A hosted VLM call costs far more than a local OCR pass, and the page leaves your infrastructure. Pages per $1 drop sharply when every page goes to a model.
Using one well
- Route, don’t blanket. Most pages of a born-digital PDF have a faithful PDF text layer that is exact and free. Spend VLM calls on scans, charts, handwriting and complex layouts. This is page routing. DocSlurp’s
autoquality sends only pages that need it;goodsends every page;fastsends none. - Cross-check when you can. If the page has a text layer or an OCR pass, compare word counts and numbers. A VLM output that lost half the words, or contains numbers the page does not, is suspect. DocSlurp can optionally run a second, cost-capped vision pass that corrects first-pass Markdown against the page image, controlled by an operator flag.
- Keep the page reference. Store the page number with every block so a reader can check the transcription against the image. Citations that open the cited page turn an invisible error into a visible one.
When to reach for it
Use a VLM when the page’s meaning lives in its pixels: scans and photos, charts, diagrams, forms, handwriting, math-heavy pages and layouts that defeat rule-based extraction. For clean digital text, the text layer is faster, cheaper and exact. The best pipelines treat the VLM as the expert you call in, not the first reader of every page.
In DocSlurp
In DocSlurp
The vision-extract stage reads selected page images with a vision model: auto sends scans and complex pages, good sends every page, fast none (core/worker/src/stages/vision-extract.ts, core/packages/shared/src/quality.ts). Prompts require verbatim Markdown with LaTeX math (core/worker/src/tools/vision-model-profiles.ts). Hosted vision is skipped when the privacy mode forbids it.