AI Engineer Roadmap All lessons Interactive Quiz GitHub

Multimodal Embeddings and CLIP

Intermediate · 11 min read · Updated · By Saurabh Pathak

In short: CLIP trains one encoder for images and one for text so that a picture and its caption land close together in the same vector space, using a contrastive loss that rewards the right pairs and punishes the wrong ones.

Play the interactive lesson: CLIP: one shared spaceRead on GitHub
On this page
  1. Explain it like I'm new
  2. Watch it
  3. How it works
  4. Code example
  5. When to use it / when not to
  6. Common mistakes
  7. Recap
  8. Interview questions
  9. Go deeper

Level: Intermediate · Reading time: 11 min · Phase: 9 · Multimodal AI

Before you start: Embeddings and Cosine Similarity. Also helpful: Temperature, because the same idea appears in the loss.

Words to know:

Explain it like I'm new

Imagine a huge map where every photo and every sentence gets a pin. You want the pin for a photo of a cat to sit right next to the pin for the sentence "a photo of a cat", and far from "a plate of pasta".

To teach that, you show the system a batch of pairs: three photos with their three captions. You ask it to score every photo against every caption. That gives a 3 x 3 table. The right answers are on the diagonal (photo 1 with caption 1, and so on). Training nudges the system so the diagonal scores go up and the other cells go down. Repeat across hundreds of millions of image-caption pairs from the web (the CLIP paper describes this at scale) and the map organises itself so that meaning, not file type, decides who sits near whom.

The payoff is search and classification with no special training. To search images with a sentence, embed the sentence and find the nearest image pins. To classify a photo, write the possible labels as sentences and see which sentence is nearest.

Watch it

CLIP: one shared space: Two encoders learn to put matching pairs close together.
CLIP: one shared space: Two encoders learn to put matching pairs close together.

It plays on a loop. Read the caption in the box, one step at a time. Want to go at your own pace? Step through it with Previous and Next.

How it works

u_i = image_encoder(image_i) / ||image_encoder(image_i)||      (length 1)
v_j = text_encoder(text_j)   / ||text_encoder(text_j)||        (length 1)
S[i][j] = u_i . v_j                                            (cosine similarity)
logits  = S / T                                                (T = temperature)
loss_image_to_text = mean_i  -log softmax(row i of logits)[i]
loss_text_to_image = mean_j  -log softmax(column j of logits)[j]
loss = (loss_image_to_text + loss_text_to_image) / 2
  1. Encode both sides. The image goes through an image encoder (such as a ViT). The text goes through a text transformer. Each produces a vector of the same length.
  2. Normalise to length 1. Then a dot product is exactly cosine similarity.
  3. Score every pair in the batch: an N x N matrix. With N pairs there are N right cells and N x N - N wrong cells, so every batch automatically supplies many "negatives" for free.
  4. Divide by temperature so the softmax is neither too flat nor too spiky.
  5. Cross-entropy both ways. Each image must pick its caption among N captions, and each caption must pick its image among N images.
  6. Update both encoders by gradient descent.

Worked example: a 3 x 3 table

Numbers from mm_clip_contrastive.py. Three toy images and their three captions (hand-made vectors in 3 dimensions) give this cosine matrix (rows images, columns captions):

caption 0caption 1caption 2
image 00.990.320.14
image 10.300.990.32
image 20.150.380.98
  1. Row 0 at temperature 0.5: divide by 0.5: logits 1.98, 0.64, 0.27. Softmax gives probabilities 0.693, 0.182, 0.125. The right answer is column 0, so the row loss is -ln(0.693) = 0.366.
  2. Do the same for every row and every column, and average: the symmetric loss is 0.395 at T = 0.5, and 0.683 at T = 1.0.
  3. Chance level. If every cell were equal, each row would give probability 1/3 to the right caption and the loss would be ln(3) = 1.099. Our table is well below that, so the pairs are well separated.
  4. Temperature again: at T = 0.07 the loss here is 0.000 to three places, because dividing by a small number makes even modest gaps decisive. Small temperatures make the loss demand clear separation; the CLIP paper describes how it handles this value.
  5. Loss falls as pairs get closer. In the demo's noise sweep, where each pair shares a hidden concept plus noise on both sides, and T = 0.1: noise 2.0 gives loss 4.819, noise 1.0 gives 3.473, noise 0.5 gives 0.707, noise 0.1 gives 0.000. Less noise means matched pairs are more alike and the loss drops.
  6. Bigger batch, harder game. With N = 32 pairs the chance loss is ln(32) = 3.466. The demo's 32-pair set (noise 0.5, T = 0.1) gets 0.165. More negatives per batch make the task harder, which is why large batches matter for this method.

Zero-shot classification. Embed each label as a prompt (for example "a photo of a cat", "a photo of a dog", "a photo of a car"), embed the image, take the highest cosine. In the demo, image 0 scores 0.99, 0.32, 0.14 and is labelled "cat"; image 1 gives "dog"; image 2 gives "car". No classifier was trained for these three labels.

Limits you should know:

flowchart LR
    IM[Images] --> IE[Image encoder]
    TX[Captions] --> TE[Text encoder]
    IE --> N1[Normalise]
    TE --> N2[Normalise]
    N1 --> S[N x N similarity matrix]
    N2 --> S
    S --> L[Contrastive loss: raise diagonal]

Code example

From mm_clip_contrastive.py (run python 11-multimodal-ai/code/mm_clip_contrastive.py):

import math

def softmax(xs):
    m = max(xs)
    e = [math.exp(x - m) for x in xs]
    return [v / sum(e) for v in e]

def info_nce(sim, temperature):
    logits = [[s / temperature for s in row] for row in sim]
    def ce(rows):
        return sum(-math.log(softmax(r)[i]) for i, r in enumerate(rows)) / len(rows)
    cols = [list(c) for c in zip(*logits)]
    return (ce(logits) + ce(cols)) / 2

In real projects you use a pretrained model: load it through a library such as sentence-transformers or the Transformers documentation, embed your images and texts, and store the vectors in a vector index as in Embeddings. For vector arithmetic see rag_vector_math.py. Always embed queries and documents with the same model.

When to use it / when not to

TaskFit
Search a photo library with a sentenceVery good
Zero-shot sorting into labels you can describe in wordsGood start, test it
Find near-duplicate or similar imagesGood
Exact counting, reading small text, strict spatial answersPoor: use OCR or a VLM
A domain far from web photosEvaluate first; fine-tuning may be needed

Common mistakes

Recap

If you remember only three things:

Interview questions

1. Explain the CLIP training objective. <details><summary>Answer</summary> For a batch of N matching image-text pairs, compute the N x N cosine similarity matrix of normalised embeddings, divide by a temperature, and apply cross-entropy so that each image picks its own caption among the N, and each caption picks its own image. The two losses are averaged. The other N - 1 items in the batch act as negatives. </details>

2. How does zero-shot image classification work with CLIP? <details><summary>Answer</summary> Write each class as a text prompt, embed the prompts and the image, and choose the class whose prompt embedding has the highest cosine similarity with the image embedding. No classifier is trained, but results depend on prompt wording and on how well the domain was covered in training. </details>

3. What does temperature do in the loss, and what is the chance-level loss for a batch of 8? <details><summary>Answer</summary> It scales the similarities before softmax: a small temperature sharpens the distribution so small gaps count more. If all scores are equal, each caption gets probability 1/8, so the loss is ln(8), about 2.08. </details>

Go deeper

Prev: Video Understanding · Next: Multimodal RAG · Back to phase: Phase 9