Skip to content

LearnRepresent

Embedding space

Vector space · Latent space

The high-dimensional space a particular embedding model maps inputs into, where nearness stands for similarity and each model has its own incompatible space.

In DocSlurpIllustrated explainer

On this page
A real embedding space, flattened to two dimensionsEight inputs embedded with all-MiniLM-L6-v2 and projected to 2D. Revenue, profit and income sit together at the top left; private jet, charter and flight sit together at the right; the question "Are we on track to beat our FY2027 earnings?" sits lower left, with EBITDA below it. Cosines from the question, in the model's 384 dimensions: revenue 0.317, income 0.301, profit 0.290, EBITDA 0.176, private jet 0.116, charter 0.098, flight 0.088. The 2D view keeps 46% of the variance, so some distances mislead: from EBITDA, the question looks closest on the map but is only fourth by cosine, and revenue, the most similar at 0.282, looks sixth.revenueprofitincomeEBITDAprivate jetcharterflightAre we on track to beat our FY2027 earnings?PCA to 2D keeps 46% of the spread. The other 54% is flattened away.SIMILAR TOquestioncosine in 384-dmap rank1. revenue0.317#32. income0.301#23. profit0.290#44. EBITDA0.176#15. private jet0.116#66. charter0.098#57. flight0.088#7Bar length: cosine on a 0 to 1 scale.Orange map rank: the flat picture disagreeswith the model about who is closer.
Neighborhoods are real, but a flat map of them lies a little. These are real vectors from all-MiniLM-L6-v2, one of DocSlurp’s local models. Finance words cluster, travel words cluster, and the question lands nearest “revenue” and “income”. Line thickness is the true cosine. Pick “EBITDA”: the question looks closest on the map, yet it is only fourth in the model’s 384 dimensions, and “revenue”, the most similar, looks sixth. Even unrelated words score above zero.

The problem: a list of numbers you can’t picture

An embedding model turns “revenue” into 384 numbers. So does “profit”, and so does “private jet”. On their own the numbers mean nothing: no position says “this is about money”. What carries meaning is how the vectors sit relative to each other. The set of all places a model can put an input is its embedding space, and learning to think about that space, without being fooled by pictures of it, is most of what you need to reason about vector search.

Nearness is meaning

The model was trained so that inputs used in similar ways land in similar directions. The result is a space with neighborhoods. Here are real cosines from all-MiniLM-L6-v2, one of DocSlurp’s local models, for single words and the question “Are we on track to beat our FY2027 earnings?”:

From Closest Next Far
revenue profit 0.717 income 0.670 private jet 0.212
private jet flight 0.517 charter 0.283 income 0.122
the question revenue 0.317 income 0.301 flight 0.088

Finance words cluster. Travel words cluster. The question lands next to “revenue” and “income”, and as far from “flight” as anything in the set. A k-nearest neighbors search for the question would return the finance cluster first, which is the whole point.

Notice “flight” and “revenue” at 0.284, nearly as close as “revenue” and “EBITDA” at 0.282. Single words are noisy inputs: both are short business nouns that show up in similar sentence positions (a flight report, a revenue report). Full sentences carry more context and separate more cleanly.

Every picture of the space is a shadow

You can’t draw 384 dimensions, so people project them down to two with PCA, t-SNE or UMAP. The figure does this with PCA for the eight inputs above. The two axes it keeps hold 46% of the spread between the points; the other 54% is flattened away, and the picture lies in exactly those directions.

Pick “EBITDA” in the figure. On the map, the question is the closest point. In the model’s real space, it is only fourth. Meanwhile “revenue”, which looks sixth closest on the map, is really the most similar of all seven. Nothing is wrong with the projection: it is doing the best a flat picture can. The lesson is that a 2D map shows clusters reliably and individual distances unreliably.

This gets worse with the nonlinear methods. t-SNE and UMAP are tuned to keep local neighborhoods intact, and in exchange they distort the sizes of clusters and the distances between them. Two clusters drawn far apart may be close in the real space; a tight blob may be spread out. Use these maps to explore and to spot outliers, never to measure.

The geometry is strange

High-dimensional space doesn’t behave like the room you’re sitting in.

  • Almost everything is nearly perpendicular. Pick two random directions in 384 dimensions and their cosine is very likely to be close to 0. Meaningful similarity shows up as a small, consistent tilt toward each other.
  • Real embeddings crowd into a cone. The opposite also happens: trained models tend to put all inputs in a narrow range of directions, so even unrelated texts score well above 0. In the table, the lowest cosine is 0.088, not −1. Where “unrelated” sits varies by model, so similarity values are only comparable within one model.
  • Directions can encode relations, loosely. Older word embeddings famously had “king − man + woman ≈ queen”. Modern sentence models have weaker versions of this. Don’t build features on vector arithmetic without testing it.

Each model has its own space

Two models that both output 384 numbers have unrelated coordinate systems. Dimension 17 in one has nothing to do with dimension 17 in the other. A query embedded with model A and searched against passages embedded with model B returns noise, and nothing in the arithmetic warns you, because the shapes match.

The same applies to settings within one model: pooling method, normalization, truncated dimensions and even weight quantization can move vectors enough to matter. A space is defined by the whole recipe, not the model name. That is why systems record exactly which recipe produced each vector; see embedding versioning.

Where it breaks

  • Neighborhoods follow usage, not truth. “Inbound flight” and “outbound flight” appear in the same kinds of sentences, so the same model scores them 0.911, even though on an itinerary they are opposites.
  • The space reflects the training data. A general model’s neighborhoods are general. In a corpus full of internal acronyms, the map may put your most important terms in the wrong place.
  • Distances near the top are crowded. Many candidates sit at almost the same cosine. Small changes in wording reorder them, which is why systems fuse vector results with keyword results and rerank the shortlist.

How to use the idea

Think of vector search as “find the neighbors of the query’s position” and most behavior becomes predictable: paraphrases work because they land nearby; part numbers fail because they don’t move the vector much; switching models breaks search because the map changed. When you need to inspect a corpus, project it to 2D to find clusters and outliers, then check any distance that matters with the real cosine.

In DocSlurp

In DocSlurp

Each project keeps one vector table per embedding model and dimension count, so vectors from different spaces are never searched together (core/packages/shared/src/vector-namespace.ts). The reuse fingerprint also records provider, normalization and weights dtype (core/worker/src/embeddings/embedding-reuse.ts).

Sources

Jump to