AI Engineer Roadmap All lessons Interactive Quiz GitHub

Incremental Indexing

Intermediate · 8 min read · Updated · By Saurabh Pathak

In short: Incremental indexing updates your RAG index by re-embedding only the chunks that are new or changed, and deleting chunks that disappeared, instead of rebuilding everything.

Play the interactive lesson: Incremental indexingRead on GitHub
On this page
  1. Explain it like I'm new
  2. Watch it
  3. How it works
  4. Code example
  5. When to use it / when not to
  6. Common mistakes
  7. Recap
  8. Interview questions
  9. Go deeper

Level: Intermediate · Reading time: 8 min · Phase: 2 — Advanced RAG

Before you start: Embeddings, Chunking and Vector Databases. You should know that documents are split into chunks, and each chunk is turned into a vector and stored.

Explain it like I'm new

If one page of a 500-page book changes, you don't reprint the whole book. You swap that page. Incremental indexing does the same for your vector index.

Embedding every chunk costs time and money (API calls or GPU time). A full re-index of 1 million chunks every night is wasteful when only 200 changed. Worse, if you only add new versions without removing old ones, your index fills with stale chunks: outdated text that still gets retrieved and leads to wrong answers.

The usual trick is a content hash: a short fingerprint computed from the text (for example SHA-256). Same text → same hash. Change one character → completely different hash. By comparing hashes from the last run with the current ones, you know exactly what to add, skip and delete.

Words to know:

Watch it

Incremental indexing: Re-embed only what changed, delete what disappeared.
Incremental indexing: Re-embed only what changed, delete what disappeared.

It plays on a loop. Read the caption in the box, one step at a time. Want to go at your own pace? Step through it with Previous and Next.

How it works

  1. Chunk each document deterministically (same input → same chunks).
  2. Compute hash = sha256(chunk_text) for every chunk.
  3. Compare with the stored record of doc_id → [hashes]:
    • hash already stored → skip, no embedding call
    • new hash → embed and insert
    • old hash no longer present → delete (unless another document still uses it)
  4. Save the new doc_id → [hashes] record.
  5. For deleted documents, remove all their chunks.
Old version              New version              Action
"Intro paragraph."  ──▶  "Intro paragraph."       skip    (same hash)
"Install with pip." ──▶  "Install with uv."       embed new + delete old
"Run the server."   ──▶  "Run the server."        skip
                                                  embed calls: 1 instead of 3

Worked example

A small README has three paragraphs. Here is what the three upsert calls in the code below do (real output):

RunWhat you give itAddedUnchangedRemovedTotal embed calls
1first version3003
2same text again0303
3"pip" changed to "uv"1214

Run 1: every hash is new, so all 3 chunks are embedded. Run 2: all 3 hashes are already stored, so nothing is embedded and the call count stays at 3. Run 3: only the "Install" paragraph has a new hash. It is embedded (call 4), and the old "pip" chunk is removed so it can never be retrieved again. A full rebuild would have cost 3 more calls instead of 1.

Code example

From adv_incremental_index.py. CountingEmbedder counts calls, so the test can prove work was skipped.

from adv_incremental_index import CountingEmbedder, IncrementalIndex

emb = CountingEmbedder()
idx = IncrementalIndex(emb)
v1 = "Intro paragraph.\n\nInstall with pip.\n\nRun the server."
v2 = "Intro paragraph.\n\nInstall with uv.\n\nRun the server."

print(idx.upsert("readme", v1), emb.calls)  # {'added': 3, 'unchanged': 0, 'removed': 0} 3
print(idx.upsert("readme", v1), emb.calls)  # {'added': 0, 'unchanged': 3, 'removed': 0} 3
print(idx.upsert("readme", v2), emb.calls)  # {'added': 1, 'unchanged': 2, 'removed': 1} 4

The core of upsert:

for piece in chunk(text):
    h = content_hash(piece)
    new_hashes.append(h)
    if h not in self.vectors:          # only embed what we have never seen
        self.vectors[h] = self.embedder.embed(piece)
        self.texts[h] = piece
        added += 1
old = set(self.doc_chunks.get(doc_id, []))
self.doc_chunks[doc_id] = new_hashes
removed = self._garbage_collect(old - set(new_hashes))  # drop stale chunks nobody uses

In real projects LangChain has an indexing API with a "record manager" that tracks hashes and supports cleanup modes for deleting stale documents, and LlamaIndex has ingestion pipelines with document stores that skip unchanged documents. Check LangChain and LlamaIndex for current usage.

When to use it / when not to

StrategyCost per runStale chunks
Append onlyLowAccumulate (bad)
Full rebuildHighNone
Hash-based incrementalLowNone, if deletes are handled

Common mistakes

Recap

If you remember only three things:

Interview questions

1. How do you avoid re-embedding the whole corpus on every update? <details><summary>Answer</summary> Hash each chunk's content (and important metadata), keep a record of hashes per document, embed only hashes not yet stored, and delete hashes that disappeared. </details>

2. When must you do a full re-index anyway? <details><summary>Answer</summary> When the embedding model changes, when chunking parameters change, or when the vector schema/metadata format changes. Vectors from different models or chunkers are not compatible. </details>

3. Why can stale chunks be dangerous? <details><summary>Answer</summary> They can be retrieved and cited as current truth, producing wrong answers (old prices, revoked policies) and, if permissions changed, they may leak content to users who should no longer see it. </details>

Go deeper

Prev: API RAG · Next: Streaming Retrieval · Back to phase: Phase 2