Caching
Keeping the result of an expensive step, such as a query embedding or a list of nearest neighbors, so a repeat request can skip the work.
In DocSlurpGlossary entry
On this page
A cache stores the result of expensive work so a repeat request can skip it. In a search service the expensive steps are easy to name: embedding the query (a network call to a model provider), the vector lookup, and generating an answer.
Embedding is the clearest win. A user asks “When does the London office open?”, then refines a filter and asks again. The query text hasn’t changed, so its vector hasn’t either. A cache keyed on the text returns the vector in microseconds instead of a provider round trip, and costs nothing in tokens.
The key is the whole problem
A cache entry is only safe if its key includes everything that could change the answer:
- Who is asking. Without the tenant in the key, one customer’s cached result can be served to another, and even a cache hit can leak information as a timing difference. See multi-tenancy.
- Which model and settings. The same text embedded by two models, or at two dimensions, gives two incompatible vectors.
- Which version of the data. A cached hit list is stale the moment a document is added or replaced. Including the index version in the key makes every write invalidate old entries automatically.
In DocSlurp
Query embeddings are cached in memory per organization, keyed by provider, model, dimensions, normalization and the whitespace-collapsed query text, which is stored only as a hash. By default it holds 1,000 entries for an hour. A cache of vector search results also exists, keyed by organization, each table’s version and every query parameter, but it is off by default. Reuse of document embeddings at ingestion is incremental indexing; model-side prompt caching is separate again.
The gotcha
An in-memory cache is per process. With several server instances, hit rates are lower than a single-instance test suggests, and a deploy empties it. Measure the hit rate in production before counting on it.
In DocSlurp
In DocSlurp
Query embeddings are cached in memory per organization, keyed by provider, model, dimensions and the normalized text, 1,000 entries for an hour by default (core/server/src/search/embedding-cache.ts). A vector-result cache keyed by org, table version and every query parameter exists but is off by default (core/server/src/search/result-cache.ts).