Embedding
Dense vector · Text embedding
A list of numbers a model assigns to a piece of text, image or audio, chosen so that inputs with similar meaning get vectors that point in similar directions.
In DocSlurpIllustrated explainer
On this page
The problem: the answer uses different words
Someone asks “Are we on track to beat our FY2027 earnings?” The passage that answers it, in growth-plan-2027.pdf, says: “Revenue is projected at $48.2M for FY2027, twelve percent ahead of plan.” The two share one content word, “FY2027”. Keyword search sees a weak match. A person sees the answer immediately, because “ahead of plan” is what “on track to beat” means and revenue is what earnings are built on.
An embedding is how a computer gets some of that judgment. It replaces each text with a list of numbers chosen by a model, so that texts with similar meaning get similar lists.
How it works: text in, arrow out
An embedding model is a neural network, usually a transformer, trained on huge numbers of text pairs that should match (a question and its answer, a title and its article) and pairs that shouldn’t. Training nudges the model until matching pairs come out close together and mismatched pairs come out far apart.
At ingest you run each passage through the model once:
- The tokenizer splits the text into tokens.
- The transformer produces one vector per token, each informed by the tokens around it.
- Pooling squeezes those into one vector, for example by averaging them (mean pooling) or keeping the first token’s vector (CLS pooling).
- The result is scaled to length 1, so only its direction carries information.
The output is a vector of fixed length: 384 numbers for all-MiniLM-L6-v2, 1,024 for BGE-M3. Think of it as an arrow in a 384-dimensional space. At query time the question goes through the same model, and search becomes geometry: find the passage arrows that point closest to the question’s arrow, measured by cosine similarity.
A worked example with real numbers
These are real cosines from all-MiniLM-L6-v2 (quantized to 8 bits, mean pooling, normalized; this is DocSlurp’s cheapest-local profile) between the question and five passages:
| Passage | Cosine | Angle |
|---|---|---|
growth-plan-2027.pdf p. 12: “Revenue is projected at $48.2M for FY2027, twelve percent ahead of plan.” |
0.576 | 54.8° |
q3-earnings-memo.pdf: “Q3 earnings beat plan, with gross margin at 71 percent.” |
0.423 | 65.0° |
board-update-2026.pdf: “The board approved a $12M budget for the London expansion.” |
0.270 | 74.3° |
deal-room-guide.pdf: “The Larkspur Studio acquisition closes in the first quarter of 2027.” |
0.241 | 76.1° |
invoice-4471.pdf: “Invoice 4471 for March hosting is due within thirty days.” |
0.185 | 79.3° |
Three things to notice. The growth plan wins even though the memo contains the exact words “earnings” and “beat”: the model rewards meaning over overlap. The ordering is what you would want: the FY2027 projection, then the latest results, then the board’s budget, then deal news and billing. And the numbers are bunched: the best match sits at 55° and the invoice, which has nothing to do with the question, at 79°. Real embedding spaces are crowded, and unrelated text rarely scores near zero. Rankings are meaningful; the raw value is not a percentage of relevance.
What the vector does and doesn’t hold
No single number in the vector means “revenue” or “travel”. Meaning is spread across all the dimensions at once, and only the direction of the whole arrow is interpretable, by comparing it with other arrows. Two consequences:
- Vectors from different models can’t be compared, even when both have 1,024 numbers. Each model invents its own coordinate system. See embedding versioning.
- You can’t edit an embedding to “add” a fact. If the passage changes, embed it again.
Where it breaks
- Negation and small words. “Q3 revenue beat the forecast.” and “Q3 revenue did not beat the forecast.” score 0.880 with the same model. The word that flips the meaning barely moves the arrow.
- Exact identifiers. “Book tail N750MC for Friday.” and “Book tail N705MC for Friday.” score 0.939. Embeddings are bad at telling part numbers, error codes and names apart, which is exactly what keyword search is good at. That is the case for hybrid search.
- Length limits. Models read a bounded number of tokens. Text past the limit is cut off or rejected, so an oversized chunk is embedded from its first part only. Chunking exists partly for this reason.
- Similar is not true. A passage that confidently states the wrong revenue forecast points the same way as the right one. Similarity finds candidates; it can’t check them.
- Domain drift. A general model may not know that “NRR” and “net revenue retention” are the same, or that “TEB” means Teterboro. Evaluate on your own questions before trusting a leaderboard.
When to use embeddings
Use them whenever users describe what they want in their own words: questions, symptoms, paraphrases, other languages. Pair them with keyword search for names, codes and numbers, and put a reranker on the merged shortlist when precision matters. Embed passages once at ingest with the same model you will use for queries, store which model that was, and re-embed everything when you change it.
In DocSlurp
In DocSlurp
The embed stage embeds every chunk (core/worker/src/stages/embed.ts). The default profile is Perplexity pplx-embed-v1-0.6b (1,024 dimensions) through OpenRouter; local Transformers.js models run in a worker-thread pool by default (core/worker/src/stages/local-embedding-thread.ts). Unchanged inputs reuse earlier vectors by fingerprint.