Skip to content

LearnOperate

Incremental indexing

Updating a search index by processing only what changed, such as new files, edited files or the stages a settings change affects, instead of rebuilding everything.

In DocSlurpGlossary entry

On this page

Incremental indexing updates a search index by doing only the work that changed. Re-upload a 300-page manual with one corrected page and a naive pipeline re-parses, re-chunks and re-embeds all 300 pages. An incremental one notices that 299 pages produce the same chunks as before and reuses their results.

Content addressing makes reuse safe

The trick is to key results by their inputs, not by file or document ID. An embedding depends only on the exact text sent to the model plus the model’s settings: provider, model name, dimensions, normalization. Hash those together and you have a key that is identical whenever the vector would be identical:

key = sha256(model settings + exact chunk text)

Before calling the embedding provider, look up each chunk’s key. Hits reuse the stored vector; only misses cost tokens. The same key also catches duplicates across files, such as a standard clause shared by fifty contracts.

Settings changes are incremental too

A new answer style touches nothing in the index, a new summary prompt needs only the summary stage, and a new embedding model invalidates every vector. Classifying each change by the stages it affects keeps re-indexing proportional.

In DocSlurp

The embed stage keys each chunk by a hash of its final input and every parameter that changes the vector, then reuses vectors from the document’s previous version or from other embedding artifacts in the same organization. Configuration changes are classified as search-only, new-uploads only, re-index recommended or re-index required, and a required re-index runs only the minimal stages. Until a document finishes re-indexing, search keeps serving it from its previous index configuration, so there is no downtime.

The gotcha

Reuse is only as correct as the key. Leave out one parameter, such as a changed chunk-context prefix or embedding dimensions, and stale vectors get served as fresh ones. When in doubt, put it in the key. See also embedding versioning and document versioning.

In DocSlurp

In DocSlurp

The embed stage keys each chunk by a hash of its final text and every vector-changing parameter, and reuses vectors from earlier artifacts in the same organization (core/worker/src/embeddings/embedding-reuse.ts). Settings changes queue only the minimal stages, and documents keep serving from their old index until each re-index finishes (core/docs/config-change-jobs.md).

Sources

Jump to