AI Engineer Roadmap All lessons Interactive Quiz GitHub

Speculative Decoding

Advanced · 9 min read · Updated · By Saurabh Pathak

In short: Speculative decoding lets a small, fast draft model guess several tokens ahead, then has the big model check them all in one pass, so you get the big model's exact output faster.

Play the interactive lesson: Speculative decodingRead on GitHub
On this page
  1. Explain it like I'm new
  2. Watch it
  3. How it works
  4. Code example
  5. Common mistakes
  6. Recap
  7. Interview questions
  8. Go deeper

Level: Advanced · Reading time: 9 min · Phase: 10 · Frontier AI Engineering

Before you start: Inference Serving and Batching and KV Cache.

Explain it like I'm new

Picture a senior editor and an intern. The editor is slow and careful. The intern is quick but makes mistakes. Instead of the editor writing every word, the intern drafts the next four words, and the editor reads all four at a glance, crosses out the first wrong one, and carries on from there. If the intern is usually right, the text comes out much faster, and because the editor approves every word, the result is what the editor would have written.

That is speculative decoding. The key fact: checking four tokens costs the big model about the same as generating one, because it reads them in parallel.

Watch it

Speculative decoding: A cheap draft guesses, the big model checks in one pass.
Speculative decoding: A cheap draft guesses, the big model checks in one pass.

It plays on a loop. Read the caption in the box, one step at a time. Want to go at your own pace? Step through it with Previous and Next.

How it works

1. Why decoding is slow

Generating text is one token per big-model pass, and each pass has to load the model's weights from memory. The hardware is often waiting for memory, not busy computing. That leaves spare compute, and speculative decoding uses it (Leviathan et al., 2022; Chen et al., 2023).

2. The loop

  1. The draft model proposes k next tokens, one at a time (cheap).
  2. The target model scores all k positions in a single pass.
  3. Keep the longest prefix the target agrees with. At the first disagreement, take the target's own token instead and stop.
  4. Even if every draft is accepted, the target adds one extra token. So each round produces between 1 and k + 1 tokens.

With a correct acceptance rule (rejection sampling), the output has exactly the same distribution as the target model alone. It is not an approximation.

3. The maths

If each draft token is accepted with probability a, independently, the expected tokens per target pass is a geometric sum:

E = 1 + a + a² + ... + a^k = (1 − a^(k+1)) / (1 − a)

From the demo, with k = 4:

Acceptance aExpected tokens per passSimulated (50,000 rounds)
0.51.9381.941
0.72.7732.770
0.94.0954.092

4. Real speed-up includes the draft's cost

If one draft step costs a fraction c of a target step, a round costs k × c + 1 target-steps. Speed-up is expected tokens divided by that. With c = 0.1 and k = 4:

Acceptance aSpeed-upBest k
0.51.38x2
0.71.98x4
0.92.93x10

A bad draft backfires: a = 0.3 with a draft costing 40 percent of the target gives 0.55x, slower than not using it at all.

5. What makes a good draft

6. When it helps

Worked example

From real output of fa_speculative.py.

  1. A draft that is right 70 percent of the time, proposing 4 tokens a round.
  2. Expected tokens per target pass: 2.773 (simulation: 2.770).
  3. Draft costs 10 percent of a target step, so a round costs 1.4 steps. Speed-up: 2.773 / 1.4 = 1.98x.
  4. Swap in a weak draft (a = 0.3, cost 0.4): 0.55x. A fast guesser that is often wrong is worse than nothing.

Code example

def expected_tokens_per_pass(a, k):
    return k + 1 if a == 1 else (1 - a ** (k + 1)) / (1 - a)

def speedup(a, k, draft_cost):
    return expected_tokens_per_pass(a, k) / (k * draft_cost + 1)

print(round(expected_tokens_per_pass(0.7, 4), 3))   # 2.773
print(round(speedup(0.7, 4, 0.1), 2))                # 1.98

Run python 12-frontier-ai/code/fa_speculative.py; tests: pytest 12-frontier-ai/code/test_fa_speculative.py.

In real projects you turn on speculative decoding in your serving engine and pick a draft model. Benchmark tokens per second on your own prompts, because acceptance depends on the text.

Common mistakes

Recap

If you remember only three things:

Interview questions

1. Why is checking k tokens about as cheap as generating one? <details><summary>Answer</summary> Decoding is limited by loading the weights from memory, not by arithmetic, so scoring k positions in one pass uses spare compute at almost the same time as scoring one position. The big model reads all the draft tokens in parallel, as it does with a prompt. </details>

2. Does speculative decoding reduce output quality? <details><summary>Answer</summary> No, when implemented with the proper acceptance rule. Draft tokens are accepted or replaced so that the final text follows exactly the target model's distribution. Only latency changes. </details>

3. A team reports a 0.8x speed-up after enabling it. What do you suspect? <details><summary>Answer</summary> A low acceptance rate or an expensive draft: perhaps a mismatched model family, a draft that is too large, or a batch size so high that no spare compute existed. Measure acceptance rate, draft cost and tokens per second, and shorten k or change the draft. </details>

Go deeper

Prev: Context Engineering and Prompt Caching · Next: Preference Tuning: RLHF and DPO · Back to phase: Phase 10