AI Engineer Roadmap All lessons Interactive Quiz GitHub

KV Cache

Advanced · 9 min read · Updated · By Saurabh Pathak

In short: The KV cache stores the key and value vectors of tokens already processed, so each new token only computes its own instead of recomputing the whole prefix.

Play the interactive lesson: KV cacheRead on GitHub
On this page
  1. Explain it like I'm new
  2. Watch it
  3. How it works
  4. Code example
  5. When to use it / when not to
  6. Common mistakes
  7. Recap
  8. Interview questions
  9. Go deeper

Level: Advanced · Reading time: 9 min · Phase: 3 · LLM Concepts

Before you start: Self Attention and Decoder Only Models.

Words to know:

Explain it like I'm new

Imagine writing a long email and, before typing each new word, re-reading the entire email from the start. That is what a naive model does: to pick token 101, it reprocesses tokens 1-100 again. Then for token 102, it re-reads 1-101.

A KV cache is like keeping notes. After reading a word once, you write down what you need about it. For the next word, you just read your notes plus the new word.

Why keys and values? In self-attention, the new token's query must be compared to the keys of all earlier tokens and then mix their values. Thanks to the causal mask, earlier tokens never change when new ones are added, so their K and V stay the same. We can safely compute them once and keep them.

Watch it

KV cache: Remember each token's keys and values so they are never recomputed.
KV cache: Remember each token's keys and values so they are never recomputed.

It plays on a loop. Read the caption in the box, one step at a time. Want to go at your own pace? Step through it with Previous and Next.

How it works

  1. Prefill. Run the whole prompt through the model in one parallel pass. Save K and V for every token, in every layer and every head.
  2. Decode step. For the new token only: compute its Q, K, V. Append K and V to the cache.
  3. Attend: the new Q against all cached K, mix all cached V.
  4. Produce the next token, repeat from step 2.

Worked example

Generate 4 tokens and count how many K,V pairs get computed.

stepwithout cachewith cache
11 (token 1)1 (token 1)
22 (tokens 1-2)1 (token 2 only)
33 (tokens 1-3)1 (token 3 only)
44 (tokens 1-4)1 (token 4 only)
total1+2+3+4 = 104

The real demo prints the same pattern at bigger sizes: 10 tokens cost 55 without the cache and 10 with it.

Counting work. Generating n tokens:

tokens nwithout cachewith cache
105510
1005,050100
1,000500,5001,000
step 1: [K1 V1]
step 2: [K1 V1][K2 V2]              <- only K2,V2 are new
step 3: [K1 V1][K2 V2][K3 V3]        <- only K3,V3 are new
         \___________ reused from cache ___________/

The cost: memory. Cache size per token is roughly:

2 (K and V) × layers × kv_heads × head_dim × bytes_per_number

Example with made-up but realistic sizes: 32 layers, 8 KV heads, head_dim 128, 16-bit numbers (2 bytes): 2 × 32 × 8 × 128 × 2 = 131,072 bytes ≈ 128 KB per token. At 8,000 tokens that is about 1 GB for one conversation. That is why long contexts and many users are memory-bound, and why GQA/MQA exist.

Related ideas:

Code example

From llm_decoding.py:

def count_kv_computations(num_tokens, use_cache):
    computed = 0
    for t in range(1, num_tokens + 1):
        computed += 1 if use_cache else t   # cache: only the new token; no cache: whole prefix
    return computed

for n in (10, 100, 1000):
    print(n, count_kv_computations(n, False), count_kv_computations(n, True))
# 10 55 10
# 100 5050 100
# 1000 500500 1000

In real projects the Hugging Face generate() method uses the cache by default (use_cache=True in the model config). Serving engines such as vLLM manage it for you.

When to use it / when not to

Common mistakes

Recap

If you remember only three things:

Interview questions

1. Why can we cache K and V but not Q? <details><summary>Answer</summary> With causal attention, earlier tokens' K and V never change when new tokens arrive, and future queries need them. A query is only used once, by its own token, at its own step, so there is nothing to reuse. </details>

2. What is the main downside of the KV cache? <details><summary>Answer</summary> Memory. It grows linearly with sequence length, layers, KV heads, and batch size, and often limits how many requests or how long a context a GPU can serve. </details>

3. Name two techniques that reduce KV-cache memory. <details><summary>Answer</summary> Grouped-query or multi-query attention (fewer K/V heads), quantizing the cache to fewer bits, paged memory management (PagedAttention), or sliding-window attention that drops old tokens. </details>

Go deeper

Prev: Encoder Decoder Models · Next: Context Window · Back to phase: Phase 3