Skip to content

LearnGenerate & answer

RAG

Retrieval-augmented generation

Answering a question by first retrieving relevant passages from your documents, then asking a language model to write an answer from those passages, with citations.

In DocSlurpIllustrated explainer

On this page
Retrieval-augmented generation: ask, retrieve, answer with citationsA person asks "Are we on track to beat FY2027 earnings?". Retrieval pulls three passages from the index: [1] growth-plan-2027.pdf page 12, "FY2027 revenue is projected at $48.2M, up 31% year over year and 12% ahead of plan"; [2] growth-plan-2027.pdf page 14, "Operating margin is projected at 24% against a 20% target"; [3] q3-earnings-memo.pdf page 2, "EBITDA guidance is raised from $9.1M to $10.6M". The model answers "Yes, comfortably" in three claims, each ending in a citation marker wired back to the passage it came from.1 · retrieve2 · assemble the top 33 · answer with citationsAre we on track tobeat FY2027 earnings?project index[1] growth-plan-2027.pdf · p.12FY2027 revenue is projected at$48.2M, up 31% year over yearand 12% ahead of plan.[2] growth-plan-2027.pdf · p.14Operating margin is projected at24% against a 20% target.[3] q3-earnings-memo.pdf · p.2EBITDA guidance is raised from$9.1M to $10.6M.MODEL ANSWER1Yes, comfortably. FY2027revenue is projected at$48.2M, 12% ahead of plan.2Operating margin lands at24% vs a 20% target.3EBITDA guidance is up from$9.1M to $10.6M.
The model reads, it doesn’t remember. Retrieval picks three passages from your documents; the model writes from those alone, and every claim carries a marker that leads back to its slip. If the right slip never reaches the model, no amount of model quality can put it back.

The problem: a fluent model that has never read your plan

Someone on the finance team asks a chat model, “Are we on track to beat our FY2027 earnings?” The model has never seen growth-plan-2027.pdf or q3-earnings-memo.pdf. It still answers, confidently, with how earnings tend to go at a typical growing company: seasonality, hiring costs, maybe a growth rate it picked because similar text online used it. Every sentence sounds right. None of it is your company’s numbers.

You could fine-tune the model on your documents, but that is slow, expensive, and has to be redone every time the plan changes. Worse, a fine-tuned model still answers from memory, so it can’t tell you where an answer came from.

Retrieval-augmented generation (RAG) takes another route: don’t make the model remember your documents. Look up the relevant passages at question time, hand them to the model with the question, and ask it to answer from those passages and cite them.

How it works: retrieve, assemble, generate

1. Retrieve. Run the question against a search index built from your documents, usually hybrid search over chunks, often followed by reranking. Keep the top few passages. For our question, three come back:

# Source Passage
1 growth-plan-2027.pdf p. 12 “FY2027 revenue is projected at $48.2M, up 31% year over year and 12% ahead of plan.”
2 growth-plan-2027.pdf p. 14 “Operating margin is projected at 24% against a 20% target.”
3 q3-earnings-memo.pdf p. 2 “EBITDA guidance is raised from $9.1M to $10.6M.”

2. Assemble. Build a prompt with rules first, then the numbered passages with their sources, then the question:

Answer using only the excerpts below. Cite each claim with [n].
If the excerpts are insufficient, say so.

[1] growth-plan-2027.pdf, page 12: FY2027 revenue is projected at $48.2M ...
[2] growth-plan-2027.pdf, page 14: Operating margin is projected at 24% ...
[3] q3-earnings-memo.pdf, page 2: EBITDA guidance is raised from $9.1M to $10.6M.

Question: Are we on track to beat our FY2027 earnings?

Everything here must fit the model’s context window, with room left for the answer.

3. Generate. The model writes:

Yes, comfortably. FY2027 revenue is projected at $48.2M, 12% ahead of plan [1], and operating margin lands at 24% against a 20% target [2]. EBITDA guidance was raised from $9.1M to $10.6M [3].

Each [n] maps back to a passage, so the interface can link it to the exact page. The reader no longer has to trust the model. They can click and check.

That is the whole idea. The model’s job changes from knowing to reading. It is an open-book exam: the student doesn’t need to have memorized the textbook, only to find the right page and summarize it faithfully.

Why this split works

Search and language models are good at different things. Search is fast, cheap, exact about names and numbers, and its results can be traced to a source, but it returns passages, not answers. Language models are good at reading several passages and writing one coherent reply, but they invent facts when they don’t have them. RAG uses each for its strength.

It also gives you control you can’t get from a model alone:

  • Freshness. Update the index and the next answer reflects the change. No retraining.
  • Scope. The model sees only passages from documents the user may read, so tenant and permission boundaries are enforced by retrieval.
  • Accountability. Citations make every claim checkable.

Where it breaks

  • Retrieval misses the answer. If the passage with the revenue forecast never reaches the prompt, the model can’t use it. It either abstains or, worse, fills the gap from general knowledge. Most bad RAG answers are retrieval failures, so measure retrieval separately, with metrics like recall@k.
  • The question needs facts you learn along the way. “Who approves the jet for the exec leading the London launch?” needs the executive’s name from one document before you can search another. One retrieval can’t do that; agentic RAG can.
  • Passages disagree. An older memo says the margin target is 20%, the newer plan says 22%. The model may pick one silently or merge them. Good prompts ask it to call out conflicts; good indexes drop superseded versions.
  • The model ignores or bends the evidence. Grounding instructions are requests. A model can still add a claim no passage supports, or misquote one it cites. See hallucination.
  • The documents talk back. Retrieved text goes straight into the prompt, so a document containing instructions can try to steer the model. See prompt injection.
  • Too much context hurts. Stuffing in thirty passages to be safe costs more, and models attend less to the middle of long inputs. See lost in the middle.

When to use it

Use RAG when answers must come from a specific, changing body of documents and readers need to verify them: internal docs, contracts, manuals, support knowledge bases, research archives. Use it even with long-context models. Fitting a whole corpus in one prompt is rarely possible at scale, and costs more per question than retrieving five good passages.

Skip it when the task needs no outside facts (rewriting, formatting) or when the corpus is small and stable enough to place in the prompt whole. If questions routinely chain several lookups, start with an agentic loop rather than a single retrieval.

In DocSlurp

In DocSlurp

Chat retrieves with hybrid search, assembles cited excerpts and generates an answer; by default a bounded search agent may search several times first (core/server/src/services/project-chat.ts, core/server/src/services/chat-mcp-agent.ts, core/server/src/services/aggregate-rag.ts). Without a model key it returns the passages instead. See chat.

Sources

Jump to