Skip to content

LearnRank & assemble

Lost in the middle

Language models use information at the start and end of a long prompt more reliably than information buried in the middle.

In DocSlurpIllustrated explainer

On this page
Lost in the middle, and the reorder that puts the best passages at the edgesA schematic U-shaped curve of answer accuracy against the position of the answer passage in the prompt: highest at position 1, lowest in the middle, and high again at the last position. This is the shape reported by Liu et al. (2023) for multi-document question answering, not DocSlurp data. Below, eight passages ranked 1 to 8 are reordered for the prompt as 1, 3, 5, 7, 8, 6, 4, 2, so the two best passages sit at the first and last positions and the weakest sit in the middle.answer accuracy, higher is better12345678position in the promptCurve: the shape reported by Liu et al. (2023) for multi-document QA. Schematic, not DocSlurp data.Packed for the prompt best at the edges, weakest in the middle#1#3#5#7#8#6#4#2#1#2#3#4#5#6#7#8Ranked by score what the reranker produced
If the model reads the edges hardest, put the best passages there. Liu et al. found models used a passage most reliably near the start or the end of the prompt, and least in the middle. Packing eight ranked passages as 1, 3, 5, 7, 8, 6, 4, 2 puts #1 first and #2 last, and leaves #7 and #8 in the dip. The curve is a schematic of the published shape, not a DocSlurp measurement.

The problem: the answer was in the prompt, and the model missed it

You retrieve twenty passages for “How far above our margin target are we?” and paste them all into the prompt. The answer, from q3-earnings-memo.pdf, is passage 10. The model replies that the documents don’t state a margin target.

Move the same passage to position 1, change nothing else, and the model answers correctly. Retrieval did its job both times. What changed was where the evidence sat in the prompt.

What the research measured

Liu et al. (2023), “Lost in the Middle: How Language Models Use Long Contexts”, tested this directly. They gave models a question and a set of retrieved documents (10, 20 or 30), exactly one of which contained the answer, and moved that document through every position.

Accuracy traced a U. It was highest when the answer was at the start of the context, high again when it was at the end, and lowest in the middle. For GPT-3.5-Turbo with 20 documents, accuracy with the answer in the middle fell below its accuracy with no documents at all. They saw the same shape on a synthetic task that only asked the model to find a key in a list of key-value pairs. Models with extended context windows were not reliably better at using what was in them.

The figure’s curve is a schematic of that published shape, not a DocSlurp measurement. The size of the dip varies by model, prompt format and context length, and newer models may behave differently. The paper does not settle the cause.

The fix: put the best passages at the edges

If you can’t make the context short, you can choose the order. Rank passages by relevance, then place them alternately at the front and the back, so the strongest sit at the two edges and the weakest end up in the middle.

For eight ranked passages:

Prompt position 1 2 3 4 5 6 7 8
Passage rank #1 #3 #5 #7 #8 #6 #4 #2

The rule: walk the ranked list, putting ranks 1, 3, 5, 7 at the next free slot from the front and ranks 2, 4, 6, 8 at the next free slot from the back. #1 lands first, #2 last, and #7 and #8 meet in the dip. Membership doesn’t change, only placement.

Better fixes, in order

Reordering is the cheapest step, not the strongest:

  1. Send fewer passages. A good reranker and a small top-k shrink the middle until there is little to lose. Eight good passages beat twenty mixed ones.
  2. Send shorter passages. Contextual compression keeps only the sentences that bear on the question.
  3. Then order what’s left so the best evidence is at the edges.

Where it breaks

  • It trusts the ranking. If the true answer is ranked 7th, packing puts it in the dip, exactly where it is least likely to be used. Reordering amplifies a good ranking and punishes a bad one.
  • It confuses humans. People reading a result list expect best first. A packed list puts the second-best result last. Use relevance order for anything a person scans, and packed order only for the prompt.
  • Citation numbers follow prompt order. If passages are labeled [1] to [8] in packed order, [8] may be the second-best source. That is fine for the model, but don’t read the labels as a ranking.
  • The effect is model-specific. Measure on your model with your prompt before assuming the U, or its size.

In DocSlurp

lostInTheMiddleReorder in core/server/src/search/position-reorder.ts implements the alternating rule above; for five passages it produces 1, 3, 5, 4, 2. It runs after reranking whenever post-processing runs (reranking, MMR or compression is on, or resultOrder is "prompt") and resultOrder is not "relevance" (core/server/src/search/session-execution.ts).

Because reranking is on by default, packed order is the default, and it is also the order search returns. Position scores (1 / (1 + index)) are assigned after the reorder, so the second-best result comes back last with the lowest score. Pass resultOrder: "relevance" when you want a ranked list, for example in a results UI. The search reference describes both orders. Chat retrieves at most 12 passages per answer, which keeps the middle short.

When to use it

Pack passages when a model reads a long list and you can’t shorten it further. Keep relevance order for anything a person reads, and for evaluation, so ranking metrics like nDCG measure the ranking and not the packing.

In DocSlurp

In DocSlurp

When post-processing runs (reranking is on by default) and resultOrder is not "relevance", the selected results are packed best-first and second-best-last (core/server/src/search/position-reorder.ts). That order is also the order search returns, with position scores reassigned afterward. Pass resultOrder: "relevance" to keep ranked order. See result order.

Sources

Jump to