AI Engineer Roadmap All lessons Interactive Quiz GitHub

Multi Query Retrieval

Intermediate · 8 min read · Updated · By Saurabh Pathak

In short: Multi-query retrieval asks an LLM to rewrite one question into several different search queries, runs them all, and merges the results.

Play the interactive lesson: Multi-query retrievalRead on GitHub
On this page
  1. Explain it like I'm new
  2. Watch it
  3. How it works
  4. Code example
  5. When to use it / when not to
  6. Common mistakes
  7. Recap
  8. Interview questions
  9. Go deeper

Level: Intermediate · Reading time: 8 min · Phase: 2 — Advanced RAG

Before you start: Query Expansion and Hybrid Search. You should know that a query is sent to a retriever, and that result lists from different searches can be combined.

Explain it like I'm new

If you lose your keys, you don't search one way. You check "where did I last drive", "which jacket was I wearing", "did I leave them in the door". Each angle finds different places. Together they cover much more ground.

A single user question is one angle. If it is phrased vaguely, or the documents phrase things differently, retrieval misses. Multi-query retrieval uses a large language model (LLM) to write, say, three alternative versions of the question. Each version goes to the retriever separately. You then merge the result lists into one ranking.

Words to know:

The most common way to merge is Reciprocal Rank Fusion (RRF). It ignores raw scores (which are hard to compare across queries) and looks only at positions. Documents that rank well in many lists win.

Watch it

Multi-query retrieval: One question becomes several, and the results are merged.
Multi-query retrieval: One question becomes several, and the results are merged.

It plays on a loop. Read the caption in the box, one step at a time. Want to go at your own pace? Step through it with Previous and Next.

How it works

  1. Prompt an LLM: "Write 3 different search queries for this question, one per line."
  2. Keep the original question plus the 3 variants.
  3. Retrieve top k for each query.
  4. Merge with RRF: for each document, score = Σ 1 / (60 + rank) over every list it appears in. The constant 60 is the usual default; it softens the gap between rank 1 and rank 2.
  5. Take the top results and generate the answer with the original question.
flowchart TD
    Q[Question] --> L[LLM writes 3 variants]
    L --> Q1[Query 1] & Q2[Query 2] & Q3[Query 3]
    Q --> R0[Retrieve]
    Q1 --> R1[Retrieve]
    Q2 --> R2[Retrieve]
    Q3 --> R3[Retrieve]
    R0 & R1 & R2 & R3 --> F[RRF merge] --> T[Top k]

RRF worked example with k=60:

DocRanks in listsRRF score
A1, 31/61 + 1/63 = 0.0164 + 0.0159 = 0.0323
B2, 1, 11/62 + 1/61 + 1/61 = 0.0489
C31/63 = 0.0159

B wins: it appears in three lists near the top.

Worked example

The repo demo runs the question "Why does my pod crash over and over?" with a fake LLM that returns fixed variants. So there are four queries, each searched separately (real output):

QueryResult list
original questiond2, d3
why does my pod restartd2, d3
CrashLoopBackOff meaningd2
liveness probe fails container restartd1, d4

Merge with RRF (k=60), adding 1/(60+rank) per list:

Final top 3: ['d2', 'd3', 'd1']. The single plain query returned only ['d2', 'd3'], so the extra variant found d1.

Code example

From adv_query_transform.py. FakeLLM returns fixed variants so it runs offline.

def rrf_merge(rankings, k=60):
    scores = {}
    for ranking in rankings:
        for rank, doc_id in enumerate(ranking, start=1):
            scores[doc_id] = scores.get(doc_id, 0.0) + 1.0 / (k + rank)
    return sorted(scores, key=lambda d: (-scores[d], d))

def multi_query_retrieve(question, llm, corpus=CORPUS, k=3):
    prompt = f"Write 3 different search queries that would help answer this question, one per line.\nQuestion: {question}"
    variants = [l.strip() for l in llm.complete(prompt).splitlines() if l.strip()]
    queries = [question] + variants
    rankings = [keyword_search(q, corpus, k=k) for q in queries]
    return queries, rrf_merge(rankings)[:k]

Run python adv_query_transform.py in 02-advanced-rag/code to see the merged ranking.

In real projects LangChain ships a MultiQueryRetriever that wraps any retriever and an LLM, and LlamaIndex has fusion retrievers that do the same with RRF. Module paths change between versions, so copy the import from the current docs: LangChain, LlamaIndex.

When to use it / when not to

Single queryMulti-query + RRF
LLM calls before retrieval01
Retrieval calls1N+1
Recall on vague questionsLowerHigher

Common mistakes

Recap

If you remember only three things:

Interview questions

1. Why use RRF instead of summing similarity scores? <details><summary>Answer</summary> Scores from different queries or different retrievers (BM25 vs vectors) live on different scales. RRF uses only ranks, so it is scale-free, simple, and robust without tuning. </details>

2. What is the cost of multi-query retrieval? <details><summary>Answer</summary> One LLM call to generate variants plus N extra retrievals, which adds latency and spend. Retrievals can run in parallel; the LLM call is usually the main added delay. </details>

3. How is multi-query different from query decomposition? <details><summary>Answer</summary> Multi-query writes alternative phrasings of the same question. Decomposition splits a complex question into distinct sub-questions that may each need separate answers, which are then combined. </details>

Go deeper

Prev: Query Expansion · Next: HyDE · Back to phase: Phase 2