AI Engineer Roadmap All lessons Interactive Quiz GitHub

Multi Modal RAG

Advanced · 8 min read · Updated · By Saurabh Pathak

In short: Multi-modal RAG retrieves not just text but also images, tables, charts, audio or video, and gives them to a model that can understand them.

Play the interactive lesson: Multi-modal RAGRead on GitHub
On this page
  1. Explain it like I'm new
  2. Watch it
  3. How it works
  4. Code example
  5. When to use it / when not to
  6. Common mistakes
  7. Recap
  8. Interview questions
  9. Go deeper

Level: Advanced · Reading time: 8 min · Phase: 2 — Advanced RAG

Before you start: What is RAG, Embeddings, and Multi Vector Retriever. You should know that RAG finds relevant text and hands it to an LLM, and that embeddings turn content into searchable numbers.

Explain it like I'm new

Ask a mechanic, "Which bolt holds the alternator?" The answer is a diagram in the manual, not a paragraph. A text-only search would never find it, because the picture has no words.

Real documents mix modalities (types of data): text, images, tables, slides, scanned PDFs, screenshots, audio, video. Text-only RAG throws away anything that isn't plain text, or mangles it (tables flattened into word soup). Multi-modal RAG keeps them.

There are three common designs:

  1. Convert to text: caption images, transcribe audio, turn tables into Markdown, then use normal text RAG. Simple, but loses visual detail.
  2. Shared embedding space: use a model like CLIP that maps images and text into the same vector space, so a text query can find an image directly.
  3. Summary + original: embed a text summary or caption for search, but send the original image to a vision-language model (VLM), a model that accepts images in its input. This is the multi-vector pattern applied to images.

Words to know:

Watch it

Multi-modal RAG: Retrieve images, tables and charts, not only text.
Multi-modal RAG: Retrieve images, tables and charts, not only text.

It plays on a loop. Read the caption in the box, one step at a time. Want to go at your own pace? Step through it with Previous and Next.

How it works

  1. Parse documents into elements: text blocks, tables, figures, page images (layout-aware parsers or OCR).
  2. Represent each element: text as-is, tables as Markdown or HTML, images as captions and/or image embeddings.
  3. Index with metadata: type, page, doc_id, and a link to the original file.
  4. Retrieve with a text query (or an image query).
  5. Generate with a VLM: pass the question, retrieved text, and the original images.

Worked example

A car manual has three pieces. The question is: "which bolt holds the alternator and what torque". This is design 3 from above (the code below runs it).

  1. Parse: the manual becomes three elements: a diagram (p3-fig1), a safety sentence (p3-txt), and a torque table (p7-tbl).
  2. Represent: each gets a text caption. The diagram's caption is "Exploded diagram of the alternator showing the two mounting bolts and belt tensioner."
  3. Retrieve: the code counts words shared between the question and each caption. The diagram and the torque table share the most words (both mention the alternator and a bolt). The safety sentence shares almost none, so it is left out.
  4. Generate: the app attaches the original image manual/p3_fig1.png and the table text to the prompt for the VLM, which reads the picture and answers.

The caption was only the search handle. The model still sees the real picture.

flowchart LR
    PDF[PDF / slides] --> P[Parser]
    P --> T[Text blocks]
    P --> TB[Tables → Markdown]
    P --> IM[Images]
    IM --> CAP[Caption via VLM]
    T & TB & CAP --> V[(Vector index<br/>+ link to original)]
    Q[Question] --> V --> R[Hits]
    R --> VLM[Vision-language model<br/>gets text + original images]

Code example

A stdlib sketch of design 3: search over captions, return the original asset path to send to a VLM.

import re

ELEMENTS = [
    {"id": "p3-fig1", "type": "image", "path": "manual/p3_fig1.png",
     "caption": "Exploded diagram of the alternator showing the two mounting bolts and belt tensioner."},
    {"id": "p3-txt", "type": "text", "path": None,
     "caption": "Disconnect the battery before any electrical work."},
    {"id": "p7-tbl", "type": "table", "path": "manual/p7_torque.md",
     "caption": "Torque table: alternator bolt 45 Nm, tensioner bolt 25 Nm."},
]

def words(t):
    return set(re.findall(r"[a-z0-9]+", t.lower()))

def retrieve(question, k=2):
    q = words(question)
    ranked = sorted(ELEMENTS, key=lambda e: -len(q & words(e["caption"])))
    return ranked[:k]

for hit in retrieve("which bolt holds the alternator and what torque"):
    print(hit["type"], hit["id"], "->", hit["path"] or "inline text")

The generation step would attach manual/p3_fig1.png as an image and the torque table as text in the prompt.

In real projects you might use CLIP-style models via sentence-transformers (which can embed images and text into one space) or a vector database's multi-modal support, plus a VLM that accepts image input. Check model cards and provider docs for exact image input formats.

When to use it / when not to

DesignCostKeeps visual detail
Convert to textLowNo
Shared embeddings (CLIP)MediumFor search, yes
Caption + original to VLMHigherYes, at answer time

Common mistakes

Recap

If you remember only three things:

Interview questions

1. Name three ways to make images searchable in RAG. <details><summary>Answer</summary> Caption them (or OCR them) and embed the text; embed them directly with a joint image–text model like CLIP; or embed a summary while storing the original and passing it to a vision-language model at answer time. </details>

2. How do you handle tables? <details><summary>Answer</summary> Extract them with a layout-aware parser, keep structure as Markdown or HTML, optionally embed an LLM-written summary for search, and pass the full table to the model. </details>

3. What's the main cost risk? <details><summary>Answer</summary> Ingest-time captioning calls per image and answer-time image tokens. Limit images per answer, downscale, and cache captions. </details>

Go deeper

Prev: GraphRAG · Next: SQL RAG · Back to phase: Phase 2