AI Engineer Roadmap All lessons Interactive Quiz GitHub

BM25

Intermediate · 10 min read · Updated · By Saurabh Pathak

In short: BM25 is a classic keyword-ranking formula that scores a document higher when it contains the query's words often, when those words are rare across the collection, and when the document is not padded with extra text.

Play the interactive lesson: BM25Read on GitHub
On this page
  1. Explain it like I'm new
  2. Watch it
  3. How it works
  4. Code example
  5. When to use it / when not to
  6. Common mistakes
  7. Recap
  8. Interview questions
  9. Go deeper

Level: Intermediate · Reading time: 10 min · Phase: 1 — RAG Foundations

Before you start: Tokenization and Hybrid Search. You should know that text is split into small pieces called tokens (here, roughly words) before it is searched.

Explain it like I'm new

Imagine searching a pile of letters for "volcano eruption". A letter that says "volcano" five times is probably more relevant than one that says it once. But a letter that says "the" fifty times tells you nothing, because every letter has "the". And a 30-page letter that mentions "volcano" twice is less focused than a postcard that mentions it twice.

BM25 turns those three intuitions into a score:

  1. Term frequency (TF): how often the query word appears in the document. More is better, but with diminishing returns — the 10th mention adds much less than the 1st. This is called saturation.
  2. Inverse document frequency (IDF): rare words matter more. "Kubernetes" in a cooking corpus is a strong signal; "the" is not.
  3. Length normalization: long documents naturally contain more words, so their counts are scaled down relative to the average length.

"BM" stands for "Best Matching"; 25 is the version number from the original research series (Okapi BM25). It powers the default ranking in Lucene, Elasticsearch and OpenSearch. It needs no training and no GPU.

Words to know:

Watch it

BM25: A classic keyword score: frequent in this document, rare overall.
BM25: A classic keyword score: frequent in this document, rare overall.

It plays on a loop. Read the caption in the box, one step at a time. Want to go at your own pace? Step through it with Previous and Next.

How it works

For a query with terms t and a document d:

score(q, d) = Σ over t in q   IDF(t) × tf(t,d) × (k1 + 1)
                              ---------------------------------------------
                              tf(t,d) + k1 × (1 − b + b × |d| / avgdl)

IDF(t) = ln( (N − df(t) + 0.5) / (df(t) + 0.5) + 1 )

Worked example

Take N = 3 documents ("cat dog", "cat bird", "cat fish"). The query term "fish" appears in 1 document, "cat" appears in all 3.

"fish" carries about 7× more weight than "cat". Now saturation, with k1 = 1.5, b = 0 (no length effect): the TF part is tf × 2.5 / (tf + 1.5). For tf = 1 → 1.0; tf = 2 → 1.43; tf = 10 → 2.17. Ten mentions score barely double one mention.

Now the repo demo. It has five short documents, and the query is "rotate refresh token". BM25 returns [(1.4727, 0)]. Only document 0 ("Refresh tokens belong in an httpOnly cookie...") shares a whole word with the query, namely "refresh". "rotate" and "token" do not match, because the document says "rotated" and "tokens" and this demo does no stemming.

flowchart LR
  Q[Query terms] --> T[For each term]
  T --> I[IDF: rare = more weight]
  T --> F[TF with saturation k1]
  T --> L[Length normalization b]
  I --> S[Multiply and sum over terms]
  F --> S
  L --> S
  S --> R[Rank documents by score]

Code example

From code/rag_bm25.py:

import math, re
from collections import Counter

def tokenize(text):
    return re.findall(r"[a-z0-9]+", text.lower())

class BM25:
    def __init__(self, texts, k1=1.5, b=0.75):
        self.k1, self.b = k1, b
        self.tfs = [Counter(tokenize(t)) for t in texts]
        self.lengths = [sum(tf.values()) for tf in self.tfs]
        self.n = len(texts)
        self.avgdl = sum(self.lengths) / self.n
        self.df = Counter(term for tf in self.tfs for term in tf)

    def idf(self, term):
        d = self.df.get(term, 0)
        return math.log((self.n - d + 0.5) / (d + 0.5) + 1)

    def score(self, query, i):
        tf, length, s = self.tfs[i], self.lengths[i], 0.0
        for t in tokenize(query):
            f = tf.get(t, 0)
            if f:
                denom = f + self.k1 * (1 - self.b + self.b * length / self.avgdl)
                s += self.idf(t) * f * (self.k1 + 1) / denom
        return s

bm = BM25(["cat dog", "cat bird", "cat fish"])
print(round(bm.idf("fish"), 2), round(bm.idf("cat"), 2))   # 0.98 0.13

The repo version also removes stop words, supports filters and returns sorted results.

In real projects (pip install rank_bm25):

from rank_bm25 import BM25Okapi

corpus = [doc.lower().split() for doc in texts]
bm25 = BM25Okapi(corpus, k1=1.5, b=0.75)
scores = bm25.get_scores("rotate refresh token".split())
top = bm25.get_top_n("rotate refresh token".split(), texts, n=3)

At scale, use Elasticsearch/OpenSearch (BM25 is their default similarity) or a database with built-in full-text search.

When to use it / when not to

Common mistakes

Recap

If you remember only three things:

Interview questions

1. What are the three ideas inside BM25? <details><summary>Answer</summary> Term frequency with saturation (k1), inverse document frequency (rare terms weigh more), and document length normalization (b). </details>

2. What do k1 and b do? <details><summary>Answer</summary> k1 controls how quickly repeated occurrences stop adding score. b controls how much longer-than-average documents are penalized; b = 0 disables length normalization. </details>

3. Why is BM25 still used in modern RAG systems? <details><summary>Answer</summary> It is fast, needs no training, is easy to explain, and handles exact terms and rare tokens that embedding models can miss. It is a strong baseline and the sparse side of hybrid search. </details>

Go deeper

Prev: Hybrid Search · Next: Sparse Retrieval · Back to phase: Phase 1