AI Engineer Roadmap All lessons Interactive Quiz GitHub

Semantic Chunking

Intermediate · 8 min read · Updated · By Saurabh Pathak

In short: Semantic chunking starts a new chunk wherever the meaning of the text shifts, by comparing the embeddings of neighbouring sentences.

Play the interactive lesson: Semantic chunkingRead on GitHub
On this page
  1. Explain it like I'm new
  2. Watch it
  3. How it works
  4. Code example
  5. When to use it / when not to
  6. Common mistakes
  7. Recap
  8. Interview questions
  9. Go deeper

Level: Intermediate · Reading time: 8 min · Phase: 1 — RAG Foundations

Before you start: Embeddings and Recursive Chunking. You should know that an embedding is a vector for meaning, and what a chunk is.

Explain it like I'm new

Imagine listening to a podcast and writing chapter markers. You do not add a marker every five minutes. You add one when the hosts change topic, from "travel tips" to "camera gear". Semantic chunking places chunk borders the same way: where the topic changes, not where a character counter says so.

Semantic means "about meaning". To measure meaning the method uses embeddings: each sentence becomes a vector, and the similarity between neighbouring sentences tells you whether they talk about the same thing. When the similarity suddenly drops, that is a breakpoint, and a new chunk begins.

The result is chunks that each cover one idea. They match questions more cleanly and contain less unrelated text. The cost is that you must embed every sentence during ingestion, and chunk sizes become uneven.

Words to know:

Watch it

Semantic chunking: Start a new chunk where the meaning of the text shifts.
Semantic chunking: Start a new chunk where the meaning of the text shifts.

It plays on a loop. Read the caption in the box, one step at a time. Want to go at your own pace? Step through it with Previous and Next.

How it works

  1. Split the document into sentences.
  2. (Optional) Group each sentence with one neighbour on each side, so single short sentences do not swing the score.
  3. Embed every sentence (or group).
  4. Compute the similarity between each sentence and the next.
  5. Mark a breakpoint where similarity falls below a threshold. Common choices: a fixed value, or a percentile of all the gaps (e.g. "the biggest 5% of drops").
  6. Join the sentences between breakpoints into chunks. Optionally enforce a maximum size with a recursive split.
sentence:     S1    S2    S3    S4    S5    S6
similarity:     0.81  0.77  0.12  0.85  0.79
                              ^ drop below threshold 0.5 -> breakpoint
chunks:       [ S1  S2  S3 ] [ S4  S5  S6 ]

Worked example

The repo demo cuts its sample text into sentences, then compares each sentence with the next using word overlap (threshold 0.1). The sample has six sentences, S1 to S6. Real scores between neighbours:

PairOverlapBelow 0.1?
S1 to S20.083yes, cut
S2 to S30.0yes, cut
S3 to S40.056yes, cut
S4 to S50.053yes, cut
S5 to S60.167no, keep together

So five chunks. The last one joins the two "refresh token" sentences, which share words like "refresh" and "tokens". Real output:

Docker images are built in layers.
Each instruction in a Dockerfile adds a layer.
Multi-stage builds use several FROM lines.
Only the final stage ends up in the shipped image, so the image stays small.
Refresh tokens should live in an httpOnly cookie. Rotate refresh tokens on every use.

Word overlap is a crude stand-in for meaning. It cut S1 from S2 even though both are about Docker layers, because they share few exact words. Real embeddings would likely keep them together. That is the reason production systems use embeddings.

Code example

The demo in code/rag_chunkers.py uses Jaccard word overlap (shared words ÷ all distinct words) instead of real embeddings, so it runs offline. Swap in a real embedding similarity in production.

import re

def split_sentences(text):
    return [s for s in re.split(r"(?<=[.!?])\s+", text.strip()) if s]

def jaccard(a, b):
    wa = set(re.findall(r"[a-z0-9]+", a.lower()))
    wb = set(re.findall(r"[a-z0-9]+", b.lower()))
    return len(wa & wb) / len(wa | wb) if wa and wb else 0.0

def semantic_chunks(text, threshold=0.1, similarity=jaccard):
    sentences = split_sentences(text)
    if not sentences:
        return []
    chunks, current = [], [sentences[0]]
    for prev, sent in zip(sentences, sentences[1:]):
        if similarity(prev, sent) < threshold:      # topic changed
            chunks.append(" ".join(current)); current = [sent]
        else:
            current.append(sent)
    chunks.append(" ".join(current))
    return chunks

print(semantic_chunks("Cats purr softly. Cats sleep a lot. Kubernetes schedules pods."))
# ['Cats purr softly. Cats sleep a lot.', 'Kubernetes schedules pods.']

In real projects LangChain ships an experimental version (pip install langchain-experimental langchain-huggingface sentence-transformers):

from langchain_experimental.text_splitter import SemanticChunker
from langchain_huggingface import HuggingFaceEmbeddings

splitter = SemanticChunker(HuggingFaceEmbeddings(model_name="all-MiniLM-L6-v2"),
                           breakpoint_threshold_type="percentile")
chunks = splitter.split_text(long_text)

LlamaIndex has a similar SemanticSplitterNodeParser.

When to use it / when not to

RecursiveSemantic
Ingest costVery lowOne embedding per sentence
Chunk sizesEvenUneven
BordersFormattingTopic shifts

Common mistakes

Recap

If you remember only three things:

Interview questions

1. How does semantic chunking decide where to cut? <details><summary>Answer</summary> It embeds sentences, measures similarity between neighbours, and cuts where similarity drops below a threshold (fixed or percentile-based), meaning the topic changed. </details>

2. What are its downsides? <details><summary>Answer</summary> Higher ingestion cost (embedding every sentence), uneven chunk sizes, sensitivity to the threshold and to sentence splitting, and no guarantee of better retrieval. </details>

3. Why group a sentence with its neighbours before embedding? <details><summary>Answer</summary> Single short sentences give noisy embeddings. A small window smooths the signal so breakpoints reflect real topic changes, not one odd sentence. </details>

Go deeper

Prev: Recursive Chunking · Next: Parent Child Chunking · Back to phase: Phase 1