Semantic Chunking
In short: Semantic chunking starts a new chunk wherever the meaning of the text shifts, by comparing the embeddings of neighbouring sentences.
On this page
Level: Intermediate · Reading time: 8 min · Phase: 1 — RAG Foundations
Before you start: Embeddings and Recursive Chunking. You should know that an embedding is a vector for meaning, and what a chunk is.
Explain it like I'm new
Imagine listening to a podcast and writing chapter markers. You do not add a marker every five minutes. You add one when the hosts change topic, from "travel tips" to "camera gear". Semantic chunking places chunk borders the same way: where the topic changes, not where a character counter says so.
Semantic means "about meaning". To measure meaning the method uses embeddings: each sentence becomes a vector, and the similarity between neighbouring sentences tells you whether they talk about the same thing. When the similarity suddenly drops, that is a breakpoint, and a new chunk begins.
The result is chunks that each cover one idea. They match questions more cleanly and contain less unrelated text. The cost is that you must embed every sentence during ingestion, and chunk sizes become uneven.
Words to know:
- Semantic: about meaning.
- Similarity: a score for how alike two pieces of text are (higher means more alike).
- Breakpoint: the spot where similarity drops and a new chunk starts.
- Threshold: the cut-off score. Below it, we start a new chunk.
- Percentile: a way to set the threshold relative to all scores, for example "the lowest 5%".
- Jaccard overlap: shared words divided by all distinct words in the two texts.
Watch it
It plays on a loop. Read the caption in the box, one step at a time. Want to go at your own pace? Step through it with Previous and Next.
How it works
- Split the document into sentences.
- (Optional) Group each sentence with one neighbour on each side, so single short sentences do not swing the score.
- Embed every sentence (or group).
- Compute the similarity between each sentence and the next.
- Mark a breakpoint where similarity falls below a threshold. Common choices: a fixed value, or a percentile of all the gaps (e.g. "the biggest 5% of drops").
- Join the sentences between breakpoints into chunks. Optionally enforce a maximum size with a recursive split.
sentence: S1 S2 S3 S4 S5 S6
similarity: 0.81 0.77 0.12 0.85 0.79
^ drop below threshold 0.5 -> breakpoint
chunks: [ S1 S2 S3 ] [ S4 S5 S6 ]
Worked example
The repo demo cuts its sample text into sentences, then compares each sentence with the next using word overlap (threshold 0.1). The sample has six sentences, S1 to S6. Real scores between neighbours:
| Pair | Overlap | Below 0.1? |
|---|---|---|
| S1 to S2 | 0.083 | yes, cut |
| S2 to S3 | 0.0 | yes, cut |
| S3 to S4 | 0.056 | yes, cut |
| S4 to S5 | 0.053 | yes, cut |
| S5 to S6 | 0.167 | no, keep together |
So five chunks. The last one joins the two "refresh token" sentences, which share words like "refresh" and "tokens". Real output:
Docker images are built in layers.
Each instruction in a Dockerfile adds a layer.
Multi-stage builds use several FROM lines.
Only the final stage ends up in the shipped image, so the image stays small.
Refresh tokens should live in an httpOnly cookie. Rotate refresh tokens on every use.
Word overlap is a crude stand-in for meaning. It cut S1 from S2 even though both are about Docker layers, because they share few exact words. Real embeddings would likely keep them together. That is the reason production systems use embeddings.
Code example
The demo in code/rag_chunkers.py uses Jaccard word overlap (shared words ÷ all distinct words) instead of real embeddings, so it runs offline. Swap in a real embedding similarity in production.
import re
def split_sentences(text):
return [s for s in re.split(r"(?<=[.!?])\s+", text.strip()) if s]
def jaccard(a, b):
wa = set(re.findall(r"[a-z0-9]+", a.lower()))
wb = set(re.findall(r"[a-z0-9]+", b.lower()))
return len(wa & wb) / len(wa | wb) if wa and wb else 0.0
def semantic_chunks(text, threshold=0.1, similarity=jaccard):
sentences = split_sentences(text)
if not sentences:
return []
chunks, current = [], [sentences[0]]
for prev, sent in zip(sentences, sentences[1:]):
if similarity(prev, sent) < threshold: # topic changed
chunks.append(" ".join(current)); current = [sent]
else:
current.append(sent)
chunks.append(" ".join(current))
return chunks
print(semantic_chunks("Cats purr softly. Cats sleep a lot. Kubernetes schedules pods."))
# ['Cats purr softly. Cats sleep a lot.', 'Kubernetes schedules pods.']
In real projects LangChain ships an experimental version (pip install langchain-experimental langchain-huggingface sentence-transformers):
from langchain_experimental.text_splitter import SemanticChunker
from langchain_huggingface import HuggingFaceEmbeddings
splitter = SemanticChunker(HuggingFaceEmbeddings(model_name="all-MiniLM-L6-v2"),
breakpoint_threshold_type="percentile")
chunks = splitter.split_text(long_text)
LlamaIndex has a similar SemanticSplitterNodeParser.
When to use it / when not to
- Useful for long, loosely structured text where topics shift without headings: transcripts, meeting notes, essays.
- Less useful for well-structured docs. Headings already mark topics, and recursive chunking is cheaper.
- Measure it. Semantic chunking is not always better than a tuned recursive splitter.
| Recursive | Semantic | |
|---|---|---|
| Ingest cost | Very low | One embedding per sentence |
| Chunk sizes | Even | Uneven |
| Borders | Formatting | Topic shifts |
Common mistakes
- No maximum size, so one long on-topic section becomes a giant chunk.
- A threshold tuned on one document and reused on very different ones.
- Bad sentence splitting (abbreviations like "e.g.", code, lists) producing junk "sentences".
- Expecting it to fix poor text extraction from PDFs.
Recap
If you remember only three things:
- Semantic chunking cuts where similarity between neighbouring sentences drops below a threshold, meaning the topic changed.
- It costs one embedding per sentence and gives uneven chunk sizes, so cap the maximum size.
- The demo uses word overlap as a cheap stand-in, and it can cut where a real embedding would not.
Interview questions
1. How does semantic chunking decide where to cut? <details><summary>Answer</summary> It embeds sentences, measures similarity between neighbours, and cuts where similarity drops below a threshold (fixed or percentile-based), meaning the topic changed. </details>
2. What are its downsides? <details><summary>Answer</summary> Higher ingestion cost (embedding every sentence), uneven chunk sizes, sensitivity to the threshold and to sentence splitting, and no guarantee of better retrieval. </details>
3. Why group a sentence with its neighbours before embedding? <details><summary>Answer</summary> Single short sentences give noisy embeddings. A small window smooths the signal so breakpoints reflect real topic changes, not one odd sentence. </details>
Go deeper
- LangChain (SemanticChunker in langchain-experimental) — https://python.langchain.com/
- LlamaIndex (SemanticSplitterNodeParser) — https://docs.llamaindex.ai/
- Reimers & Gurevych, Sentence-BERT (sentence embeddings) — https://arxiv.org/abs/1908.10084
Prev: Recursive Chunking · Next: Parent Child Chunking · Back to phase: Phase 1