AI Engineer Roadmap All lessons Interactive Quiz GitHub

Multi Head Attention

Intermediate · 9 min read · Updated · By Saurabh Pathak

In short: Multi-head attention runs several smaller attention operations side by side, so different heads can track different kinds of relationships at once.

Play the interactive lesson: Multi-head attentionRead on GitHub
On this page
  1. Explain it like I'm new
  2. Watch it
  3. How it works
  4. Code example
  5. When to use it / when not to
  6. Common mistakes
  7. Recap
  8. Interview questions
  9. Go deeper

Level: Intermediate · Reading time: 9 min · Phase: 3 · LLM Concepts

Before you start: Self Attention and Transformer Architecture.

Words to know:

Explain it like I'm new

Imagine reviewing a contract with a team. One person checks names and dates, another checks money amounts, a third checks who promised what to whom. Each reads the same document but looks for different links. At the end, you combine their notes.

Multi-head attention works the same way. Instead of one big attention with vectors of width 512, the model splits the width into, say, 8 heads of width 64. Each head has its own Q, K, V weight matrices, so it can learn its own idea of "relevant". Research has found heads that, for example, attend to the previous token or link pronouns to nouns, but there is no fixed rule about what a head learns.

The cost is roughly the same as one big head, because each head works on a smaller slice.

Watch it

Multi-head attention: Several small attentions side by side, each tracking something different.
Multi-head attention: Several small attentions side by side, each tracking something different.

It plays on a loop. Read the caption in the box, one step at a time. Want to go at your own pace? Step through it with Previous and Next.

How it works

  1. Start with token vectors X of shape n × d_model.
  2. Project into Q, K, V with learned matrices (each d_model × d_model).
  3. Split each of Q, K, V into h heads: shape n × (d_model / h) per head.
  4. Run scaled dot-product attention in each head independently.
  5. Concatenate the head outputs back to n × d_model.
  6. Multiply by an output matrix W_o to mix information across heads.
X (3 tokens x 4 features)
        | split width into 2 heads
   +----+----+
 head 1    head 2        each: 3 x 2
 attention attention     (own Q,K,V)
   +----+----+
        | concat -> 3 x 4
        | x W_o
      output (3 x 4)

Numbers from the original paper: d_model = 512, h = 8, so each head has d_k = d_v = 64.

Variants you will see in modern LLMs:

Worked example

Take 3 tokens with 4 numbers each and 2 heads (the same X as the code example below).

  1. Split. Each token's 4 numbers become two pairs. Head 1 gets the first pair of every token: [1,0], [0,1], [1,1]. Head 2 gets the last pair: [1,0], [0,1], [0,0].
  2. Attend separately. Each head builds its own 3 × 3 table of weights. Head 1's first row is [0.4, 0.2, 0.4]. Head 2's first row is [0.5, 0.25, 0.25]. They differ because the heads see different features.
  3. Odd case. In head 2, token 3 is [0, 0], so all its scores are 0 and it spreads attention evenly: [0.33, 0.33, 0.33].
  4. Merge. Each head returns 3 × 2 numbers. Glued together they are 3 × 4 again, the same shape as the input.
  5. Mix. In a real model, W_o then lets the two heads' results combine.

Code example

From llm_attention.py. Q = K = V = X here (no learned projections) so the split is easy to see.

from llm_attention import split_heads, merge_heads, multi_head_attention

X = [[1.0, 0.0, 1.0, 0.0],
     [0.0, 1.0, 0.0, 1.0],
     [1.0, 1.0, 0.0, 0.0]]

heads = split_heads(X, num_heads=2)
print(heads[0])   # [[1.0, 0.0], [0.0, 1.0], [1.0, 1.0]]  -> first 2 features of each token
print(heads[1])   # [[1.0, 0.0], [0.0, 1.0], [0.0, 0.0]]  -> last 2 features
assert merge_heads(heads) == X

out, weights = multi_head_attention(X, num_heads=2)
print(len(weights))  # 2 separate attention tables, one per head

Notice the heads see different features and so produce different attention tables. That is the whole point.

In real projects (PyTorch):

import torch
mha = torch.nn.MultiheadAttention(embed_dim=512, num_heads=8, batch_first=True)
x = torch.randn(2, 10, 512)          # (batch, tokens, features)
out, attn_weights = mha(x, x, x)     # self-attention
print(out.shape)                     # torch.Size([2, 10, 512])

When to use it / when not to

VariantK/V headsKV cache sizeTypical use
Multi-head (MHA)hlargestoriginal transformer
Grouped-query (GQA)g (1 < g < h)mediummany current open LLMs
Multi-query (MQA)1smallestfast decoding

Common mistakes

Recap

If you remember only three things:

Interview questions

1. Why use multiple heads instead of one big one? <details><summary>Answer</summary> Each head has its own projections, so different heads can attend to different positions and relationships at the same time. One head would average them into a single pattern. The compute cost is about the same because each head is narrower. </details>

2. What do GQA and MQA change, and why? <details><summary>Answer</summary> They let several query heads share the same key and value heads (all heads in MQA, groups in GQA). This shrinks the KV cache and memory bandwidth during decoding, which speeds up generation. </details>

3. If d_model is 4096 and there are 32 heads, what is the head dimension? <details><summary>Answer</summary> 4096 / 32 = 128. </details>

Go deeper

Prev: Self Attention · Next: Positional Encoding · Back to phase: Phase 3