Skip to content

LearnIndex & store

Proposition indexing

Dense X retrieval

Rewriting passages into short, self-contained factual statements with an LLM and indexing each statement on its own.

In DocSlurpGlossary entry

On this page

Proposition indexing replaces passages with propositions: short, self-contained statements, each expressing one fact in its own words. A language model rewrites every paragraph, resolving pronouns and adding missing subjects, and each proposition is embedded and searched on its own. The idea comes from Dense X Retrieval (Chen et al., 2023), which found that retrieving at proposition granularity can beat retrieving whole passages or sentences.

From page 2 of travel-desk-guide.pdf:

The executive assistant books the Learjet through the travel desk, and it is wheels-up within three hours of the request. For trips over $40k, it also needs the CFO’s pre-approval.

becomes:

The executive assistant books the Learjet through the travel desk.
The Learjet is wheels-up within three hours of the executive assistant's request.
A Learjet trip over $40k needs the CFO's pre-approval.

The last proposition no longer depends on “it”. Each statement matches questions about it precisely, and none can be cut badly.

The gotchas

  • It costs a model call per paragraph at ingest, and many more vectors to store.
  • The model can get it wrong. A rewrite can drop a qualifier, merge two facts or invent a connection. Keep a link from each proposition to its source paragraph and cite the source, not the rewrite.
  • Small units lose context for answering. A proposition finds the right place; the model usually needs the surrounding paragraph to answer well, which is the small-to-big pattern.

In DocSlurp the stage exists but is not yet part of retrieval: with propositionIndex on, paragraphs of at least 100 characters are atomized by a text model, embedded and stored with their source paragraph, but search in the pinned core still queries chunks only.

In DocSlurp

In DocSlurp

The optional propositionIndex stage turns paragraphs of 100+ characters into propositions with a text model, embeds them and stores them (core/worker/src/stages/proposition-index.ts). In the pinned core, search does not query propositions yet: retrieval still runs over chunks.

Sources

Jump to