Skip to content

LearnRepresent

Multimodal embeddings

Image embeddings · Cross-modal embeddings

Embeddings from a model that maps more than one kind of input, such as text and page images, into the same space, so a text query can find a picture.

In DocSlurpGlossary entry

On this page

A multimodal embedding model maps different kinds of input, such as text and images, into one shared embedding space. A photo of a business jet and the sentence “a business jet” land near each other. That lets you search images with words, or a scanned page with a question, without first converting everything to text.

CLIP made the idea mainstream by training on hundreds of millions of image and caption pairs: pull each matching pair together, push mismatched pairs apart. Newer document-focused models embed whole page screenshots, so charts, tables, stamps and handwriting contribute to the vector even when no text layer exists.

For documents, this opens a route around OCR. A scanned london-office-tour.jpg can be embedded as an image and found by “photo of the new London lobby”, even if text extraction got nothing useful. The trade-off: the match tells you which page to look at, not what it says. To quote or cite the answer you still need the text, from OCR or a vision-language model, at least for the pages that matter.

The gotchas: the query must be embedded by exactly the same model as the images, often with a specific prompt or prefix that model expects; mixing models gives silent nonsense. Embedding a page image usually costs more than embedding its extracted text, since an image becomes many more input tokens. And fine print rarely survives: a page reduced to one vector can find “the page with the revenue chart” but not a single digit in a table cell. Multi-vector page models built on late interaction do better on detail at a higher storage cost.

In DocSlurp

In DocSlurp

Opt-in Visual Pages render pages at a 1,024 px long edge and embed the images (core/worker/src/stages/page-embed.ts) with a Jina v5 omni private endpoint or Gemini Embedding 2 (core/worker/src/deferred/page-image-embedders.ts). Query text is embedded by the same model.

Sources

Jump to