Skip to content

LearnRetrieve

Query planning

Query planner · Retrieval planning · Retrieval strategy selection

Deciding, before or during retrieval, which searches to run for a question, in what order and at what depth, instead of sending every question through one fixed search.

In DocSlurpIllustrated explainer

On this page

“What is the cost code for N750MC?”

Searches
1
Model calls
0
Latency
one round trip

The answer is one passage and the question already contains its words. Anything more is wasted work.

search model call rule done at ingest

The shape of the question picks the plan. A single fact needs one search. A constraint becomes a filter. Independent parts are searched in parallel; a chain must run in order and pays for a model call between hops. A whole-corpus question can’t be answered by top-k at all. Count the searches and model calls: each plan has to earn them against simply retrieving deeper.

The problem: one search shape for every question

A retrieval pipeline is usually built around one question shape: a single fact that sits in one passage. Embed the question, run hybrid search, keep the top 8. That handles “What is the cost code for N750MC?” well.

Real questions come in other shapes, and the same pipeline fails each of them differently:

  • “Compare the margin target in the growth plan with the margin in the Q3 earnings memo” needs two passages from two documents. One search tends to fill the top results with whichever half has more matching text.
  • “Who approves the jet for the exec leading the London launch?” needs a fact you don’t have yet. No passage mentions both the launch and the approver.
  • “What do the Q3 slides say about London?” contains a constraint (“slides”) that should remove passages, not nudge a ranking.
  • “What is driving our growth heading into FY2027?” has no passage-sized answer at all.

Raising top-k helps none of these reliably. The fix is to choose a different set of searches for each shape. That choice is the query plan.

The idea: pick the searches before running them

The term comes from databases. A SQL planner reads a query, estimates how many rows each step would touch, and picks an order of index scans and joins. A retrieval planner does the same job with different operators:

Question shape Plan Extra cost
One fact One hybrid search None
Explicit constraint Extract a filter (self-query), then search inside it A rule or one model call
Several independent parts Query decomposition: one search per part, in parallel One search per part
Vague or oddly worded Query rewriting, HyDE or multi-query retrieval One model call, more searches
Specific case of a general rule Step-back prompting: search the specific and the general One model call, one search
A chain of facts Multi-hop retrieval: search, read, search again Serial searches and model calls
The whole corpus Summaries prepared at ingest (GraphRAG, RAPTOR) Paid at ingest

The figure shows the plan for each shape. Pick a question and compare how many searches and model calls it spends.

Who does the planning

There are three places to put the decision.

  • Rules. Patterns in the question trigger a plan: a quoted file name becomes a filter, “ vs “ triggers a split. Rules are fast, free and predictable, and they miss anything phrased differently.
  • A classifier. A small model sorts the question into a shape first. Adaptive-RAG trains one to choose between no retrieval, one retrieval and an iterative loop.
  • The answering model itself. In agentic RAG the model gets a search tool and plans as it goes. This is the most flexible option and the most expensive, because every step is a model call that rereads everything found so far.

Parallel or serial

Two plans with the same number of searches can feel very different. Independent searches (the parts of a compound question, a rewrite alongside the original) run in parallel, so latency stays close to one search. Dependent searches (hop two needs hop one’s result) must run in order, and each hop usually waits on a model. A plan is worth drawing as a small graph: what can start now, and what is waiting on something.

The baseline every plan must beat

A plan spends extra searches, and sometimes extra model calls. The same budget could have gone to plain depth: a larger candidate depth and a reranker. So the fair comparison for any plan is the default search at equal work, not the default search as it ships.

We learned to be strict about this. In a DocSlurp diagnostic of 384 known-passage probes, a single keyword query put the known passage in the top ten 206 times. Fusing three result lists did so 183 times, with 4 gains and 27 losses. The two arms did unequal backend work, so this isn’t a ranking of the methods, but it does show that more searches can return less.

Where it breaks

  • Misclassification. A planner that splits a question which wasn’t compound, or invents a filter nobody asked for, does worse than no planner. A wrong filter excludes the right passage outright.
  • Cost without a ceiling. Plans that loop need a step budget and a stopping rule, or they keep searching for evidence that isn’t there instead of abstaining.
  • Merging undoes the plan. Running the right sub-searches and then fusing everything into one ranked list can bury the passage that only one sub-search found.
  • Invisible plans. When an answer is wrong, the first question is which searches ran. Log the plan and the effective queries with the result.

In DocSlurp

Search has a rule-based planner. selfQuery extracts filters before retrieval. Then at most one multi-search mode applies, checked in this order: queryDecompose, stepback, iterativeMultiHop. All are off by default, so a plain search is one hybrid retrieval. Chat is different: its default search agent sees the conversation and plans its own queries within a step budget.

In DocSlurp

In DocSlurp

A rule-based planner applies at most one multi-search mode per request: queryDecompose, then stepback, then iterativeMultiHop, all off by default (server/src/search/session-planner.ts). selfQuery extracts filters first. In chat, the default search agent plans its own queries.

Sources

Jump to