Skip to content

LearnRepresent

Embedding

Dense vector · Text embedding

A list of numbers a model assigns to a piece of text, image or audio, chosen so that inputs with similar meaning get vectors that point in similar directions.

In DocSlurpIllustrated explainer

On this page
Passages become arrows; similar meanings point the same wayA question and five passages pass through the same embedding model. Each comes out as a list of 384 numbers, drawn here as an arrow of length 1. The question arrow points between the growth-plan and earnings-memo arrows, with the board-update arrow close by: these form an earnings cluster. The Larkspur deal note and the invoice point far away, forming an off-topic cluster. The layout is illustrative and keeps the real similarity order: growth plan, earnings memo, board update, deal note, invoice. The question's first four numbers are −0.007, −0.067, −0.025 and 0.026.TEXT INMODELARROWS OUT (ILLUSTRATIVE 2D)QAre we on track to beatour FY2027 earnings?Ggrowth-plan-2027.pdf · p.12FY2027: $48.2M, 12% ahead.Mq3-earnings-memo.pdfQ3 earnings beat plan.Bboard-update-2026.pdf$12M London budget approved.Ldeal-room-guide.pdfLarkspur deal closes Q1 2027.Iinvoice-4471.pdfInvoice due within thirty days.embedding modellength 1earningsoff-topicQGMBLIEach arrow is 384 numbers (all-MiniLM-L6-v2). The question starts[−0.007, −0.067, −0.025, 0.026, … 380 more]
The model never compares texts. It places each one, and similar meanings end up pointing the same way. The question shares one word with the growth plan (“FY2027”), yet its arrow lands between the plan and the earnings memo, while the invoice points elsewhere. The 2D angles are drawn for clarity and keep the real similarity order; the real angles, and the cosines, are in the table below.

The problem: the answer uses different words

Someone asks “Are we on track to beat our FY2027 earnings?” The passage that answers it, in growth-plan-2027.pdf, says: “Revenue is projected at $48.2M for FY2027, twelve percent ahead of plan.” The two share one content word, “FY2027”. Keyword search sees a weak match. A person sees the answer immediately, because “ahead of plan” is what “on track to beat” means and revenue is what earnings are built on.

An embedding is how a computer gets some of that judgment. It replaces each text with a list of numbers chosen by a model, so that texts with similar meaning get similar lists.

How it works: text in, arrow out

An embedding model is a neural network, usually a transformer, trained on huge numbers of text pairs that should match (a question and its answer, a title and its article) and pairs that shouldn’t. Training nudges the model until matching pairs come out close together and mismatched pairs come out far apart.

At ingest you run each passage through the model once:

  1. The tokenizer splits the text into tokens.
  2. The transformer produces one vector per token, each informed by the tokens around it.
  3. Pooling squeezes those into one vector, for example by averaging them (mean pooling) or keeping the first token’s vector (CLS pooling).
  4. The result is scaled to length 1, so only its direction carries information.

The output is a vector of fixed length: 384 numbers for all-MiniLM-L6-v2, 1,024 for BGE-M3. Think of it as an arrow in a 384-dimensional space. At query time the question goes through the same model, and search becomes geometry: find the passage arrows that point closest to the question’s arrow, measured by cosine similarity.

A worked example with real numbers

These are real cosines from all-MiniLM-L6-v2 (quantized to 8 bits, mean pooling, normalized; this is DocSlurp’s cheapest-local profile) between the question and five passages:

Passage Cosine Angle
growth-plan-2027.pdf p. 12: “Revenue is projected at $48.2M for FY2027, twelve percent ahead of plan.” 0.576 54.8°
q3-earnings-memo.pdf: “Q3 earnings beat plan, with gross margin at 71 percent.” 0.423 65.0°
board-update-2026.pdf: “The board approved a $12M budget for the London expansion.” 0.270 74.3°
deal-room-guide.pdf: “The Larkspur Studio acquisition closes in the first quarter of 2027.” 0.241 76.1°
invoice-4471.pdf: “Invoice 4471 for March hosting is due within thirty days.” 0.185 79.3°

Three things to notice. The growth plan wins even though the memo contains the exact words “earnings” and “beat”: the model rewards meaning over overlap. The ordering is what you would want: the FY2027 projection, then the latest results, then the board’s budget, then deal news and billing. And the numbers are bunched: the best match sits at 55° and the invoice, which has nothing to do with the question, at 79°. Real embedding spaces are crowded, and unrelated text rarely scores near zero. Rankings are meaningful; the raw value is not a percentage of relevance.

What the vector does and doesn’t hold

No single number in the vector means “revenue” or “travel”. Meaning is spread across all the dimensions at once, and only the direction of the whole arrow is interpretable, by comparing it with other arrows. Two consequences:

  • Vectors from different models can’t be compared, even when both have 1,024 numbers. Each model invents its own coordinate system. See embedding versioning.
  • You can’t edit an embedding to “add” a fact. If the passage changes, embed it again.

Where it breaks

  • Negation and small words. “Q3 revenue beat the forecast.” and “Q3 revenue did not beat the forecast.” score 0.880 with the same model. The word that flips the meaning barely moves the arrow.
  • Exact identifiers. “Book tail N750MC for Friday.” and “Book tail N705MC for Friday.” score 0.939. Embeddings are bad at telling part numbers, error codes and names apart, which is exactly what keyword search is good at. That is the case for hybrid search.
  • Length limits. Models read a bounded number of tokens. Text past the limit is cut off or rejected, so an oversized chunk is embedded from its first part only. Chunking exists partly for this reason.
  • Similar is not true. A passage that confidently states the wrong revenue forecast points the same way as the right one. Similarity finds candidates; it can’t check them.
  • Domain drift. A general model may not know that “NRR” and “net revenue retention” are the same, or that “TEB” means Teterboro. Evaluate on your own questions before trusting a leaderboard.

When to use embeddings

Use them whenever users describe what they want in their own words: questions, symptoms, paraphrases, other languages. Pair them with keyword search for names, codes and numbers, and put a reranker on the merged shortlist when precision matters. Embed passages once at ingest with the same model you will use for queries, store which model that was, and re-embed everything when you change it.

In DocSlurp

In DocSlurp

The embed stage embeds every chunk (core/worker/src/stages/embed.ts). The default profile is Perplexity pplx-embed-v1-0.6b (1,024 dimensions) through OpenRouter; local Transformers.js models run in a worker-thread pool by default (core/worker/src/stages/local-embedding-thread.ts). Unchanged inputs reuse earlier vectors by fingerprint.

Sources

Jump to