AI Engineer Roadmap All lessons Interactive Quiz GitHub

Reranking

Intermediate · 8 min read · Updated · By Saurabh Pathak

In short: Reranking is a second, more careful pass that re-orders the chunks your first search returned, so the best ones end up at the top.

Play the interactive lesson: RerankingRead on GitHub
On this page
  1. Explain it like I'm new
  2. Watch it
  3. How it works
  4. Code example
  5. When to use it / when not to
  6. Common mistakes
  7. Recap
  8. Interview questions
  9. Go deeper

Level: Intermediate · Reading time: 8 min · Phase: 2 — Advanced RAG

Before you start: Dense Retrieval and Embeddings. You should know that a first search turns text into vectors and returns the closest chunks.

Explain it like I'm new

Think of hiring. First, a recruiter skims 500 CVs in an hour and keeps 30 that mention the right keywords. Then a senior engineer reads those 30 carefully and picks the best 5. The recruiter is fast but shallow. The engineer is slow but sharp. You need both: the engineer cannot read 500 CVs, and the recruiter alone picks some poor fits.

RAG retrieval works the same way. The first stage, called retrieval or recall stage, uses a fast method (vector search, keyword search like BM25, or both) to pull, say, the top 50 candidates (possibly relevant chunks) out of millions. It is tuned to not miss good chunks, even if it lets in some noise.

The second stage, the reranker, scores only those 50 candidates with a slower but smarter model and keeps the top 3–5. Those few go into the prompt. The reranker is tuned for precision: the chunks at the top should really answer the question.

Words to know:

Why bother? The language model only sees what you put in the prompt. If the right chunk sits at position 14, it never gets used. Models also pay less attention to text buried in the middle of long prompts, so order matters too.

Watch it

Reranking: A slower, more careful second pass that fixes the order.
Reranking: A slower, more careful second pass that fixes the order.

It plays on a loop. Read the caption in the box, one step at a time. Want to go at your own pace? Step through it with Previous and Next.

How it works

  1. User asks a question.
  2. A fast retriever returns the top k candidates (often 20–100).
  3. A reranker scores each (question, chunk) pair.
  4. Sort by the new score and keep the top n (often 3–5).
  5. Build the prompt from those n chunks.
flowchart LR
    Q[Question] --> R[Fast retriever<br/>top 50]
    R --> X[Reranker<br/>scores 50 pairs]
    X --> T[Top 5]
    T --> L[LLM prompt]

Tiny worked example. The question is "How do I cancel my subscription and get a refund?"

ChunkFast scoreRerank score
"Refund refund refund. Refund policy."0.480.20
"To cancel your subscription open Settings… A refund is issued…"0.380.67

The first chunk repeats "refund", so the fast vector match likes it. The reranker checks how much of the whole question each chunk covers, and flips the order.

Worked example

Trace the four steps with the real demo corpus (adv_rerank.py, four tiny documents). The question is the same cancel-and-refund one, with k=3 candidates and top_n=2 kept.

  1. Fast retrieval scores every document by vector similarity and keeps the top 3: refund-faq 0.4811, cancel-guide 0.3849, pricing 0.1768. (shipping scores 0 and is dropped.)
  2. Rerank scores each of the 3 pairs by how much of the whole question the text covers: cancel-guide 0.6667, refund-faq 0.2, pricing 0.2.
  3. Sort and slice to 2. refund-faq and pricing tie at 0.2, and the code breaks ties by name, so pricing wins.
  4. Final prompt chunks: cancel-guide, pricing.

The chunk that really answers the question moved from 2nd place to 1st. Note that pricing only made the cut because we asked for 2; it is a weak match, which is why real systems use a score threshold too.

Code example

The repo demo adv_rerank.py does exactly this offline:

from adv_rerank import retrieve_then_rerank

candidates, final = retrieve_then_rerank("How do I cancel my subscription and get a refund?", k=3, top_n=2)
print(candidates)  # [(0.4811, 'refund-faq'), (0.3849, 'cancel-guide'), (0.1768, 'pricing')]
print(final)       # [(0.6667, 'cancel-guide'), (0.2, 'pricing')]

The core of reranking is just "score pairs, sort, slice":

def rerank(query, candidates, docs, top_n=2):
    rescored = [(cross_score(query, docs[doc_id]), doc_id) for _, doc_id in candidates]
    rescored.sort(key=lambda x: (-x[0], x[1]))
    return rescored[:top_n]

In real projects you use a trained cross-encoder model, for example with sentence-transformers:

from sentence_transformers import CrossEncoder

model = CrossEncoder("cross-encoder/ms-marco-MiniLM-L-6-v2")
scores = model.predict([(question, chunk) for chunk in candidates])
ranked = sorted(zip(scores, candidates), reverse=True)[:5]

Hosted rerank APIs exist too. They follow the same idea: send the query and candidate texts, get back scores.

When to use it / when not to

First-stage retrieverReranker
SpeedVery fast (pre-computed index)Slower (scores each pair)
ScaleMillions of chunksTens to hundreds
GoalRecall (don't miss)Precision (best on top)

Common mistakes

Recap

If you remember only three things:

Interview questions

1. Why not just use the reranker for everything? <details><summary>Answer</summary> A cross-encoder must run once per (query, document) pair at query time, and nothing can be pre-computed. Over millions of documents that is far too slow. So we use a cheap retriever to shortlist and the reranker only on the shortlist. </details>

2. Your reranker didn't improve results. What do you check? <details><summary>Answer</summary> First, whether the correct chunk is in the candidate set at all (recall@k of the first stage). If not, the reranker cannot help. Then check the candidate count, input truncation, domain mismatch of the reranker model, and whether the eval set is large enough to show a difference. </details>

3. Where does reranking sit relative to filtering by permissions? <details><summary>Answer</summary> Permission and metadata filters should apply before or during retrieval, so forbidden chunks never become candidates. Reranking only orders chunks the user is allowed to see. </details>

Go deeper

Prev: Phase 1 — RAG Foundations · Next: Cross Encoder · Back to phase: Phase 2