Vector quantization
Scalar quantization · SQ
Storing vectors with fewer bits per number, for example rounding 32-bit floats to 8-bit integers, to cut memory and speed up distance math at a small accuracy cost.
In DocSlurpIllustrated explainer
On this page
The problem: vectors are big
An embedding with 1,024 dimensions stored as 32-bit floats takes 4,096 bytes. A million chunks need about 4.1 GB just for vectors, before any index. Vector search is fastest when those bytes sit in memory or a nearby cache, and every query reads many of them. Memory, cache misses and bandwidth become the bottleneck long before arithmetic does.
But most of those 32 bits are spent on precision that ranking doesn’t need. A similarity search only has to decide which vectors are closer, not compute distances to seven decimal places.
Scalar quantization: round every number
Scalar quantization stores each coordinate with fewer bits. With 8 bits, the range of values is divided into 256 evenly spaced levels, and each float is replaced by the index of the nearest level: one byte instead of four.
Take the coordinate 0.66 with a range of −1 to 1 (the figure’s toy range; real indexes learn the range from the data).
- 4 bits, 16 levels. Levels are 2/15 ≈ 0.133 apart. 0.66 sits at position (0.66 + 1) / 2 × 15 = 12.45, rounds to level 12, and is stored as −1 + 12 × 2/15 = 0.600. Error: 0.06.
- 8 bits, 256 levels. Levels are 2/255 ≈ 0.0078 apart. 0.66 sits at 211.65, rounds to level 212, and comes back as 0.6627. Error: 0.0027.
At 8 bits no coordinate moves by more than half a level, 1/255 ≈ 0.0039 here. The vector is four times smaller.
What the error does to search
Rounding moves every stored vector slightly. Distances shift with them. The question is whether the order changes.
In the figure, the query is q = (0.62, −0.35, 0.48) and two stored vectors compete:
| Exact distance | 4-bit distance | 8-bit distance | |
|---|---|---|---|
| A = (0.70, −0.42, 0.20) | 0.2995 | 0.3238 | 0.2999 |
| B = (0.77, −0.56, 0.66) | 0.3146 | 0.2996 | 0.3180 |
A is truly nearer, by 0.015. At 4 bits the rounding errors are larger than that gap, and B looks nearer. At 8 bits the errors are under 0.004 and A stays ahead.
That is the general pattern. Quantization rarely sends a search to the wrong neighborhood, because errors are small compared with the distance between unrelated vectors. It reshuffles candidates that were nearly tied. If your top 10 and your 11th are separated by more than the quantization error, nothing changes.
Refining: compressed for the search, exact for the final order
Because the damage is limited to close calls, a common fix is to refine: search with the compressed vectors, take more candidates than you need (say 3 × k), then re-score just those with the original full-precision vectors and keep the best k. You pay for a few exact distances per query and get the exact order back for the candidates that matter.
Other kinds of quantization
- Product quantization splits each vector into sub-vectors and replaces each with a code from a learned codebook. It compresses far more (often 16 to 64 times) with larger errors, so refining matters more.
- Binary quantization keeps one bit per dimension (its sign). It is 32 times smaller and very fast to compare, and loses the most information.
- Matryoshka embeddings shrink the other axis: fewer dimensions instead of fewer bits per dimension. The two combine.
Where it breaks
- Outliers stretch the range. With one range for all values, a few extreme coordinates spread the levels apart, and typical values get coarser buckets. Per-dimension ranges or clipping help.
- Near-duplicates and tight clusters. When many candidates are almost equally close (repeated boilerplate, versions of the same document), the order among them becomes noisy. Refining or deduplication fixes it.
- Error compounds with other approximations. A graph index that misses neighbors, plus quantization that reorders them, plus a small candidate list, can add up. Measure recall with the full stack, not each piece alone.
- Keep the originals. You can’t refine, re-quantize or rebuild an index without the full-precision vectors.
When to use it
8-bit scalar quantization is close to free: four times less memory, faster distance math, and order changes only among near ties. Use it by default for large indexes. Reach for product or binary quantization when memory is the hard limit, and pair them with refining.
In DocSlurp
DocSlurp’s LanceDB index is HNSW over 8-bit scalar-quantized vectors (IVF_HNSW_SQ), built with L2 distance. Refining is opt-in: setting LANCEDB_REFINE_FACTOR to an integer from 1 to 16 makes each search re-score that many times the requested results with full-precision vectors, and raises the search breadth to match.
In DocSlurp
In DocSlurp
The vector index is HNSW over 8-bit scalar-quantized vectors (Index.hnswSq, core/worker/src/stages/lancedb-maintenance.ts). An opt-in LANCEDB_REFINE_FACTOR (1 to 16) re-scores extra candidates with full-precision vectors (core/server/src/search/lancedb-query-tuning.ts).