AI Engineer Roadmap All lessons Interactive Quiz GitHub

Inside a Transformer Block: Residuals, LayerNorm and the Feed-Forward Network

Intermediate · 11 min read · Updated · By Saurabh Pathak

In short: A transformer block is attention plus three supporting parts: residual connections that carry the signal past each step, a normalisation layer that keeps numbers in a healthy range, and a feed-forward network that processes each token on its own.

Play the interactive lesson: Inside a transformer blockRead on GitHub
On this page
  1. Explain it like I'm new
  2. Watch it
  3. How it works
  4. Code example
  5. When to use it / when not to
  6. Common mistakes
  7. Recap
  8. Interview questions
  9. Go deeper

Level: Intermediate · Reading time: 11 min · Phase: 3 · LLM Concepts

Before you start: Self Attention and Q, K and V Explained From Scratch. Attention is only half of a block, and this page covers the other half.

Explain it like I'm new

Earlier topics covered attention (self-attention, multi-head). But a real model is a tall stack of identical blocks, often 30 to 100 of them, and attention is only one ingredient in each block. Picture a factory line with many stations:

Plain definitions:

Watch it

Inside a transformer block: Attention plus three supporting parts.
Inside a transformer block: Attention plus three supporting parts.

It plays on a loop. Read the caption in the box, one step at a time. Want to go at your own pace? Step through it with Previous and Next.

How it works

Here is one block in the common modern layout, called pre-norm:

flowchart TD
    X[x: token vectors] --> N1[Norm]
    N1 --> A[Attention: tokens MIX]
    X --> P1((+))
    A --> P1
    P1 --> N2[Norm]
    N2 --> F[FFN: each token PROCESSED]
    P1 --> P2((+))
    F --> P2
    P2 --> Y[output: same shape as x]
  1. Normalise the input, then run attention. Add the result back onto the original x (first residual).
  2. Normalise again, run the FFN on every token separately. Add the result back (second residual).
  3. The output has exactly the same shape as the input, so blocks can be stacked as many times as you like.

Residual connections: a highway for signal and gradients

Imagine a layer that computes f(x) = 0.5 · x. Stack 20 layers.

DesignEach layer doesAfter 20 layersGradient factor
No residual0.5·xsignal × 0.5²⁰ ≈ 0.000001 (gone)0.5²⁰ ≈ 0.00000095
Residualx + 0.5·x = 1.5·xsignal × 1.5²⁰ ≈ 3,325 (alive)1.5²⁰ ≈ 3,325

The numbers come from depth_experiment and residual_gradient_factor in llm_block.py. Real sub-layers are not so extreme; the point is the direction. Without the residual, every layer can shrink the signal and the gradient (the learning signal sent backward during training, see topic 24) is multiplied by a small number 20 or 100 times and vanishes. With a residual, the derivative of x + f(x) is 1 + f'(x). That "1" is a highway: learning signal always has a direct path from the loss to early layers. Each layer also only has to learn a small correction to the input, which is easier than learning a whole new representation. Notice that the residual version in the toy grows fast. That is the second reason for normalisation.

LayerNorm by hand

LayerNorm takes one token's vector and forces its numbers to average 0 with variance 1. Take x = [2, 4, 6, 8].

  1. Mean = (2+4+6+8)/4 = 5.
  2. Subtract the mean: [-3, -1, 1, 3].
  3. Variance = average of squares = (9+1+1+9)/4 = 5. Standard deviation = √5 ≈ 2.236.
  4. Divide: [-1.342, -0.447, 0.447, 1.342]. Check: mean 0, variance 1.
  5. Apply learned gamma (scale) and beta (shift): out = gamma · normalised + beta. Gamma starts at 1 and beta at 0; training lets the model undo the normalisation where that helps.

A tiny eps (like 0.00001) is added to the variance so we never divide by zero.

RMSNorm is the leaner modern variant (used in Llama-style models). It skips the mean and the beta and only divides by the root mean square: for the same vector, squares are 4+16+36+64 = 120, mean 30, RMS ≈ 5.477, result [0.365, 0.730, 1.095, 1.461]. It is cheaper and works about as well in practice.

Pre-norm vs post-norm

Post-norm (original 2017 paper)Pre-norm (most modern LLMs)
Formulax = Norm(x + f(x))x = x + f(Norm(x))
Highway clean?No, the norm sits on the highwayYes, x passes untouched
Training deep stacksFragile, needs careful warm-upMuch more stable

The feed-forward network

For each token vector (width d):

  1. Expand with a matrix to 4d (a wider "scratch space").
  2. Apply a nonlinearity: ReLU (max(0, v)) in the original paper, GELU (a smooth version of ReLU) in GPT-style models, or SwiGLU in many newer ones. SwiGLU multiplies two projections together, one passed through a smooth gate, using three matrices; the hidden width is shrunk to about 8/3·d so the parameter count stays close to the classic design.
  3. Project back down to d.

Crucially, the FFN looks at one token at a time; token 5's output does not depend on token 6. So the division of labour is: attention mixes tokens, the FFN processes each token. Research on trained models suggests the FFN layers act like a big memory of facts and patterns, and they hold roughly two thirds of the parameters of the block, as the table shows.

Parameter counts at d_model = 4096

PartMatricesFormulaWeightsShare
Attention (Q, K, V, output)4 of d × d4d²67,108,86433%
FFN (4× expansion)d × 4d and 4d × d8d²134,217,72867%
Block total12d²201,326,592100%

(Biases and norm weights are tiny and ignored.) Multiply by the number of layers to get most of the model's size. For how those weights are then shrunk, see quantization.

Code example

import math

def layer_norm(x, eps=1e-5):
    mean = sum(x) / len(x)
    var = sum((v - mean) ** 2 for v in x) / len(x)
    return [(v - mean) / math.sqrt(var + eps) for v in x]

def residual_stack(depth, scale, residual):
    x = 1.0
    for _ in range(depth):
        x = x + scale * x if residual else scale * x
    return x

print(layer_norm([2, 4, 6, 8]))        # [-1.342, -0.447, 0.447, 1.342]
print(residual_stack(20, 0.5, False))  # ~0.000001  (vanished)
print(residual_stack(20, 0.5, True))   # ~3325      (survives)

The full demo, including rms_norm, gelu, ffn, block_forward (pre-norm and post-norm) and param_counts, is in llm_block.py. Run python 03-llm-concepts/code/llm_block.py; tests are in test_llm_block.py.

In real projects you use the framework's layers:

import torch.nn as nn

class Block(nn.Module):
    def __init__(self, d, heads):
        super().__init__()
        self.n1, self.n2 = nn.LayerNorm(d), nn.LayerNorm(d)
        self.attn = nn.MultiheadAttention(d, heads, batch_first=True)
        self.ffn = nn.Sequential(nn.Linear(d, 4 * d), nn.GELU(), nn.Linear(4 * d, d))

    def forward(self, x):
        h = self.n1(x)
        x = x + self.attn(h, h, h, need_weights=False)[0]   # residual 1
        return x + self.ffn(self.n2(x))                      # residual 2

When to use it / when not to

PartJobLooks across tokens?Share of block weights
Attentionmix information between tokensYes~1/3
FFNtransform and store knowledge per tokenNo~2/3
Residualcarry signal and gradientsNo0
Normkeep scale stableNotiny

Common mistakes

Recap

If you remember only three things:

Interview questions

1. Why do transformers use residual connections? <details><summary>Answer</summary> The derivative of x + f(x) is 1 + f'(x), so gradients always have a direct path back to early layers and do not vanish in deep stacks. Each layer also only learns a small correction to its input, which is easier to optimise. </details>

2. What is the difference between LayerNorm and RMSNorm, and what is pre-norm? <details><summary>Answer</summary> LayerNorm subtracts the mean, divides by the standard deviation and applies gamma and beta. RMSNorm skips the mean subtraction and beta and divides by the root mean square, which is cheaper. Pre-norm applies the norm to the input of each sub-layer (x + f(Norm(x))), keeping the residual path clean, which makes deep models train more stably than post-norm. </details>

3. What does the feed-forward network do, and how many parameters does it hold? <details><summary>Answer</summary> It expands each token vector to about 4× width, applies a nonlinearity (GELU, SwiGLU), and projects back, independently per token. Attention is 4d² weights and the FFN is 8d², so the FFN holds about two thirds of a block's parameters and is thought to store much of the model's factual knowledge. </details>

Go deeper

Prev: Q, K and V Explained From Scratch · Next: How LLMs Are Trained · Back to phase: Phase 3