AI Engineer Roadmap All lessons Interactive Quiz GitHub

Neurons and Forward Propagation

Beginner · 9 min read · Updated · By Saurabh Pathak

In short: A neuron multiplies its inputs by weights, adds them up with a bias, and passes the total through an activation function; forward propagation is just doing that layer after layer to get a prediction.

Play the interactive lesson: Neurons and forward propagationRead on GitHub
On this page
  1. Explain it like I'm new
  2. Watch it
  3. How it works
  4. Code example
  5. When to use it / when not to
  6. Common mistakes
  7. Recap
  8. Interview questions
  9. Go deeper

Level: Beginner · Reading time: 9 min · Phase: 0 · ML Foundations

Before you start: Loss and Gradient Descent (what weights are and why we adjust them).

Words to know:

Explain it like I'm new

Think of a neuron as a voter who weighs advice. Three friends tell you whether to go out tonight. You trust one friend a lot (large weight), another a little (small weight), and one gives bad advice (negative weight). You add up their weighted opinions, add your own mood (the bias), and then decide how strongly to say yes. That last squashing step is the activation function.

A single neuron can only draw a straight line between two groups. Stack neurons into layers and the network can bend and combine those lines into complicated shapes. The activation function is what makes stacking useful: without it, many layers would collapse into one big weighted sum, no more powerful than a single neuron.

Three common activations:

Watch it

Neurons and forward propagation: Weighted sum, bias, activation, layer after layer.
Neurons and forward propagation: Weighted sum, bias, activation, layer after layer.

It plays on a loop. Read the caption in the box, one step at a time. Want to go at your own pace? Step through it with Previous and Next.

How it works

z = w1*x1 + w2*x2 + ... + b        (weighted sum plus bias)
a = activation(z)                  (the neuron's output)

sigmoid(z) = 1 / (1 + e^-z)        tanh(z) = (e^z - e^-z) / (e^z + e^-z)        ReLU(z) = max(0, z)
zsigmoidtanhReLU
-20.1192-0.96400
-10.2689-0.76160
00.50000.00000
10.73110.76161
20.88080.96402

Forward pass through a 2-2-1 network (2 inputs, 2 hidden neurons, 1 output; the same tiny network used in ml_tiny_network.py, with sigmoid everywhere). The starting weights are:

hidden neuron 1: weights (0.5, -0.4), bias  0.1
hidden neuron 2: weights (0.3,  0.8), bias -0.2
output neuron  : weights (0.7, -0.6), bias  0.05

Feed in x = (0, 1):

  1. Hidden neuron 1: z = 0.5*0 + (-0.4)*1 + 0.1 = -0.3, so a = sigmoid(-0.3) = 0.4256.
  2. Hidden neuron 2: z = 0.3*0 + 0.8*1 + (-0.2) = 0.6, so a = sigmoid(0.6) = 0.6457.
  3. Output neuron: z = 0.7*0.4256 + (-0.6)*0.6457 + 0.05 = 0.2979 - 0.3874 + 0.05 = -0.0395, so a = sigmoid(-0.0395) = 0.4901.

The network says 0.4901. If the right answer for (0, 1) is 1 (as in XOR), the squared-error loss 0.5 * (0.4901 - 1)^2 is 0.1300. That is a poor prediction, as expected from untrained weights. In fact all four XOR inputs give almost the same output at the start: 0.5368, 0.4901, 0.5466, 0.5021. The job of training is to change the weights so these separate to near 0, 1, 1, 0. See Backpropagation.

In matrix form, a whole layer is one line: a = activation(W x + b), where each row of W holds one neuron's weights.

flowchart LR
    X1[x1] --> H1((hidden 1))
    X2[x2] --> H1
    X1 --> H2((hidden 2))
    X2 --> H2
    H1 --> O((output))
    H2 --> O
    O --> Y[prediction]

Code example

From ml_tiny_network.py (run python 00-ml-foundations/code/ml_tiny_network.py):

import math

def sigmoid(z):
    return 1.0 / (1.0 + math.exp(-z))

def forward(x, params):
    W1, b1, W2, b2 = params["W1"], params["b1"], params["W2"], params["b2"]
    z1 = [sum(W1[j][i] * x[i] for i in range(2)) + b1[j] for j in range(2)]
    a1 = [sigmoid(z) for z in z1]
    z2 = sum(W2[0][j] * a1[j] for j in range(2)) + b2[0]
    a2 = sigmoid(z2)
    return {"x": list(x), "z1": z1, "a1": a1, "z2": z2, "a2": a2}

In real projects you stack layers in a framework (this needs pip install torch):

import torch.nn as nn
model = nn.Sequential(nn.Linear(2, 2), nn.Sigmoid(), nn.Linear(2, 1), nn.Sigmoid())

When to use it / when not to

ActivationTypical use
ReLU (and variants such as GELU)hidden layers of most modern networks
Sigmoidoutput for a yes/no probability; gates inside some networks
Softmaxoutput for choosing one of many classes
tanhwhen you want outputs centred on zero (some recurrent networks)
None (linear)output for predicting an unbounded number

Common mistakes

Recap

If you remember only three things:

Interview questions

1. What does a single neuron compute? <details><summary>Answer</summary> It takes the weighted sum of its inputs, adds a bias, and applies an activation function: a = f(w · x + b). </details>

2. Why do we need non-linear activation functions? <details><summary>Answer</summary> Without them, each layer is a linear map and a stack of linear maps is just another linear map, so depth adds no power. A non-linearity between layers lets the network represent curved and complex functions. XOR, for example, cannot be solved by one linear neuron but can by two layers with a non-linearity. </details>

3. Why is ReLU usually preferred to sigmoid in hidden layers? <details><summary>Answer</summary> Sigmoid's slope is at most 0.25 and nearly 0 for large positive or negative inputs, so gradients shrink as they pass back through many layers (vanishing gradients). ReLU has slope 1 for positive inputs, is cheap to compute, and trains deep networks more easily. Its drawback is that a neuron stuck in the negative region outputs 0 and gets no gradient ("dying ReLU"). </details>

Go deeper

Prev: Loss and Gradient Descent · Next: Backpropagation · Back to phase: Phase 0