Multimodal embeddings
Image embeddings · Cross-modal embeddings
Embeddings from a model that maps more than one kind of input, such as text and page images, into the same space, so a text query can find a picture.
In DocSlurpGlossary entry
On this page
A multimodal embedding model maps different kinds of input, such as text and images, into one shared embedding space. A photo of a business jet and the sentence “a business jet” land near each other. That lets you search images with words, or a scanned page with a question, without first converting everything to text.
CLIP made the idea mainstream by training on hundreds of millions of image and caption pairs: pull each matching pair together, push mismatched pairs apart. Newer document-focused models embed whole page screenshots, so charts, tables, stamps and handwriting contribute to the vector even when no text layer exists.
For documents, this opens a route around OCR. A scanned london-office-tour.jpg can be embedded as an image and found by “photo of the new London lobby”, even if text extraction got nothing useful. The trade-off: the match tells you which page to look at, not what it says. To quote or cite the answer you still need the text, from OCR or a vision-language model, at least for the pages that matter.
The gotchas: the query must be embedded by exactly the same model as the images, often with a specific prompt or prefix that model expects; mixing models gives silent nonsense. Embedding a page image usually costs more than embedding its extracted text, since an image becomes many more input tokens. And fine print rarely survives: a page reduced to one vector can find “the page with the revenue chart” but not a single digit in a table cell. Multi-vector page models built on late interaction do better on detail at a higher storage cost.
In DocSlurp
In DocSlurp
Opt-in Visual Pages render pages at a 1,024 px long edge and embed the images (core/worker/src/stages/page-embed.ts) with a Jina v5 omni private endpoint or Gemini Embedding 2 (core/worker/src/deferred/page-image-embedders.ts). Query text is embedded by the same model.