LanceDB
An open-source embedded vector database built on the Lance columnar file format, able to keep its tables on local disk or in object storage such as S3.
In DocSlurpGlossary entry
On this page
LanceDB is an open-source, embedded vector database. “Embedded” means it runs as a library inside your process rather than as a separate server, like SQLite. Tables are stored in the Lance columnar format, either on local disk or in object storage such as S3, and LanceDB supports vector indexes (HNSW, IVF, product and scalar quantization), BM25 full-text indexes, scalar indexes and SQL-like filters.
Because a table is just files, writers and readers can be different processes on different machines: a worker appends rows and builds indexes, and an API server opens the same files to search them.
The gotchas
- Writes create versions and fragments. Each commit adds files. Without regular maintenance (compaction, index refresh, cleanup of old versions), queries slow down and storage grows.
- New rows are not indexed immediately. Rows written since the last index refresh are searched by a flat scan until maintenance folds them in.
- Concurrent writers can conflict. Commits to the same table from several processes need retries or a single writer.
- Object storage adds latency. Reading index pages from S3 on every query is slow; cache connections, table handles and index pages.
In DocSlurp
Each project gets its own LanceDB database per embedding model and dimension, and the API derives that location only from the authorized organization and project, never from request input. The worker writes chunk rows by merge-insert on id, through a queue that commits each table’s pending documents in batches. Background maintenance compacts the table, folds new rows into existing indexes, builds a BM25 index on chunk text, and adds the HNSW vector index and an id btree once the table reaches 1,000 rows.
In DocSlurp
In DocSlurp
DocSlurp stores chunk vectors in LanceDB, one database per project and embedding model (core/packages/shared/src/vector-namespace.ts). The worker writes rows by merge-insert on id (core/worker/src/stages/lancedb-writer.ts) and maintains HNSW, BM25 and id indexes; the server opens only the project database derived from authorized access (core/server/src/search/lancedb.ts).