Backpropagation

1. What backpropagation is

The backpropagation algorithm is probably the most fundamental component of an Artificial Neural Network. After each forward pass through a network, backpropagation performs a backward pass while adjusting the model parameters (weights and biases). The parameters are updated by computing gradients of expressions through automatic differentiation and the recursive application of the chain rule — the two machines built in Chapter 2.

The whole algorithm serves one formula:

w′ = w − η · ∂E/∂w

Geometrically, the slides picture an error surface E(w) over the weight vector w = [w1, w2]. The gradient ∇E(w) = [∂E/∂w1, ∂E/∂w2] points along the direction of steepest increase, so the minus sign in the update sends every weight downhill.

2. One neuron: the forward pass

The slides work the algorithm on the smallest interesting object: a neuron with two inputs in1 and in2, a bias, and the sigmoid as activation function. The trick that makes the mechanics visible is to expand the sigmoid into its primitive operators, so that the neuron becomes a computational graph of seven simple nodes:

in₁ × w₁  ─┐
              ├─ + ─ + w₀ ─ −x ─ eˣ ─ 1+x ─ 1/x ─ out
in₂ × w₂  ─┘

The last four nodes are just the sigmoid written out: σ(net) = 1 / (1 + e−net).

Now instantiate the example: w₀ = −3, in1 = −1, w1 = 2, in2 = −2, w2 = −3. During the forward pass, from inputs to outputs, each operator computes its output value:

3. One neuron: the backward pass

During the backward pass, from the outputs to the inputs, each operator i does two things:

1. it learns the sensibility ∂out/∂ρi of the neuron final output with respect to its own output variation, using the chain rule;
2. it computes the local gradient ∂ρi/∂x of its output with respect to its inputs.

The upstream sensibility multiplied by the local gradient is the sensibility handed to the node below. Here is the entire descent, in the order the slides walk it:

Run it yourself, one node at a time. The forward thread must finish before the backward thread can move, because the chain rule needs the stored node values.

Two structural facts fall out of the table, and both are worth memorising:

• The add operator has a constant local gradient equal to 1, so the gradients on all its inputs are equal to the gradient on its output. Addition is a gradient router: it copies.
• The multiply operator swaps its inputs, so it is a gradient scaler.

4. Collapsing the sigmoid into one operator

Expanding the sigmoid was a teaching device. In practice, grouping the simple operators composing the sigmoid function into a single operator makes the derivation more efficient. The four nodes −x, ex, 1+x, 1/x collapse into one node computing 1/(1 + e−x), whose derivative is known in closed form:

∂σ/∂x = (1 − σ(x)) · σ(x)

Evaluated at the same point: (1 − σ(1)) · σ(1) = 0.20. That is exactly the value that four chained multiplications produced in the previous section, obtained in a single step.

The rest of the backward pass is unchanged. The collapsed graph gives, in three moves:

5. The neuron as a black box

Once the internal mechanics are understood, the neuron can be treated as a black box with a clean contract:

• during the forward pass, given inputs and weights, it computes the output value out;
• during the backward pass, it computes partial derivatives on both weights and inputs.

The two formulas that define the contract, both obtained by one application of the chain rule through net:

∂out/∂wi  = (∂out/∂net) · (∂net/∂wi)  = (∂f/∂net) · ini

∂out/∂ini = (∂out/∂net) · (∂net/∂ini) = (∂f/∂net) · wi

6. A full network, one training step

The slides now assemble a network: 2 input neurons (I₁, I₂), 2 hidden neurons (H₁, H₂) and 1 output neuron (O₁), with the identity function as activation. Inputs x1 = 2, x2 = 3, desired output y = 1, learning rate η = 0.05.

Forward pass

The input is transformed layer by layer into an output and a loss, while all intermediate activations are computed and stored — storing them is not an implementation detail, it is what makes the backward pass possible.

outH1 = 2 · 0.11 + 3 · 0.21 = 0.85
outH2 = 2 · 0.12 + 3 · 0.08 = 0.48
outO1 = 0.85 · 0.14 + 0.48 · 0.15 = 0.191

The most used loss function for the regression task is the Square Error:

SE = (y − ŷ)² = (1 − 0.191)² = 0.654

To reduce the error, the predicted output needs to be increased.

Backward pass: the output neuron

We want to know how much each specific weight affects the error, so we need ∂E/∂w, computed with the chain rule. For an output neuron O_j:

∂E/∂wHi,Oj = (∂E/∂outOj) · (∂outOj/∂wHi,Oj)

with the two factors:

• ∂E/∂outOj = −2 · (y − outO1) = −2 · (1 − 0.191) = −1.62
• ∂outOj/∂wHi,Oj = (∂fO/∂netOj) · inHi,Oj = 1 · outHi, because fO is the identity function

∂E/∂wH1,O1 = −1.62 · 1 · 0.85 = −1.38
  ⇒ w′H1,O1 = 0.14 − 0.05 · (−1.38) = 0.14 + 0.07 = 0.21

∂E/∂wH2,O1 = −1.62 · 1 · 0.48 = −0.78
  ⇒ w′H2,O1 = 0.15 − 0.05 · (−0.78) = 0.15 + 0.04 = 0.19

Backward pass: the hidden neurons

For a hidden neuron H_j the same decomposition applies, but the first factor now has to be gathered from every output neuron the hidden neuron feeds:

∂E/∂wIi,Hj = (∂E/∂outHj) · (∂outHj/∂wIi,Hj)

∂E/∂outHj = ∑k=1nO ∂EOk/∂outHj

and each term of that sum is itself a chain:

∂EOk/∂outHj = ∂EOk/∂inHj,Ok = (∂EOk/∂outOk) · (∂outOk/∂inHj,Ok)
                        = (−2(yk − outOk)) · (1 · wHj,Ok)

The second factor is again simple because the activation is the identity and inIi,Hj = outIi = xi:

∂outHj/∂wIi,Hj = (∂fH/∂netHj) · inIi,Hj = 1 · xi

Putting the four numbers together:

∂E/∂wI1,H1 = −1.62 · 0.14 · 1 · 2 = −0.45  ⇒ w′ = 0.11 + 0.02 = 0.13
∂E/∂wI2,H1 = −1.62 · 0.14 · 1 · 3 = −0.68  ⇒ w′ = 0.21 + 0.03 = 0.24
∂E/∂wI1,H2 = −1.62 · 0.15 · 1 · 2 = −0.49  ⇒ w′ = 0.12 + 0.02 = 0.14
∂E/∂wI2,H2 = −1.62 · 0.15 · 1 · 3 = −0.73  ⇒ w′ = 0.08 + 0.04 = 0.12

Did it work?

Run the forward pass again with the updated weights:

outH1 = 2 · 0.13 + 3 · 0.24 = 0.98
outH2 = 2 · 0.14 + 3 · 0.12 = 0.64
outO1 = 0.98 · 0.21 + 0.64 · 0.19 = 0.327

SE = (1 − 0.327)² = 0.453     (< 0.654)

One step of backpropagation moved the prediction from 0.191 to 0.327 and the error from 0.654 to 0.453. Do it yourself below, repeatedly, and watch the loss curve fall.

7. Matrix based backpropagation

To take advantage of fast matrix libraries and powerful GPU-based computing systems, backpropagation is implemented in matrix form. This is the version every framework actually runs.

Given an ANN with L layers, let Wl (for l ∈ {2, …, L}) be the weight matrix of layer l, where the element wij contains the weight from the j-th neuron of layer l−1 to the i-th neuron of layer l; and let x be the input vector.

Forward pass, from layer 2 up to layer L

sl = Wl al−1

al = x          if l = 1
al = fl(sl)   otherwise

where sl is the vector of weighted sums (the pre-activations) of all neurons in layer l and al−1 is the vector of activations of the previous layer.

Backward pass, from layer L down to layer 2

The gradient of the error with respect to the pre-activation vector of layer l is the quantity everything else is built from:

δl = ∂E/∂sl = (∂E/∂al) ⊙ (∂al/∂sl)          if l = L
δl = (Wl+1T δl+1) ⊙ (∂al/∂sl)              otherwise

Then the gradient with respect to the weights between layer l−1 and l, and the update:

∂E/∂Wl = δl al−1T

ΔWl = −η · ∂E/∂Wl
W′l = Wl + ΔWl

8. The matrix example, worked

The slides re-run the same network in matrix form, which is a good way to check that the two formulations agree.

Forward pass, l = 2

aI = x = [2, 3]T

WH = | 0.11  0.21 |
     | 0.12  0.08 |

sH = WH aI = [0.85, 0.48]T
aH = fH(sH) = [0.85, 0.48]T

Forward pass, l = 3

WO = [ 0.14  0.15 ]

sO = WO aH = 0.191
aO = fO(sO) = 0.191

Backward pass, l = 3

∂E/∂aO = −2 · (y₁ − outO1) = −1.62
∂aO/∂sO = f′O(sO) = 1

δO = (−1.62) ⊙ 1 = −1.62

∂E/∂WO = δO aHT = −1.62 · [0.85  0.48] = [−1.38  −0.78]
ΔWO = −0.05 · [−1.38  −0.78] = [0.07  0.04]
W′O = [0.14  0.15] + [0.07  0.04] = [0.21  0.19]

Backward pass, l = 2

∂aH/∂sH = f′H(sH) = [1, 1]T

δH = (WOT δO) ⊙ [1, 1]T
   = [0.14, 0.15]T · (−1.62) ⊙ [1, 1]T = [−0.23, −0.24]T

∂E/∂WH = δH aIT = | −0.46  −0.69 |
                          | −0.48  −0.72 |

ΔWH = −0.05 · ∂E/∂WH = | 0.02  0.03 |
                            | 0.02  0.04 |

W′H = | 0.11  0.21 | + | 0.02  0.03 | = | 0.13  0.24 |
     | 0.12  0.08 |   | 0.02  0.04 |   | 0.14  0.12 |

The updated weights are identical to those obtained scalar by scalar in Section 6: 0.13, 0.24, 0.14, 0.12 for the hidden layer and 0.21, 0.19 for the output layer. Same algorithm, one written for a person and one written for a GPU.

9. Vanishing and exploding gradients

When training an ANN with backpropagation, the partial derivatives are computed backward through the network using the chain rule, by applying consecutive matrix multiplications. If the derivatives are small or large, then the gradient will decrease or increase exponentially through the model.

For shallow networks with only a few layers this is not a big problem. In deep neural networks it causes the gradient to be:

• too small to effectively train the network — the vanishing gradient problem;
• too large, resulting in a very unstable model incapable of effective learning — the exploding gradient problem.

More generally, the gradient in deep neural networks is unstable, tending to either vanish or explode in earlier layers. Because this instability is a fundamental problem for deep neural networks, it is very important to detect and solve it.

Watch the exponential behaviour directly: set a per-layer derivative and a depth, and see what survives the trip back to the first layer.

10. Gradient clipping

Gradient clipping limits the gradient to ensure optimization performs more reasonably near sharp areas of the loss surface. A common solution is to clip the gradient g by projecting it back to a ball of a given radius θ:

g′ = min(1, θ / ‖g‖) · g

Read the two cases:

Two guarantees follow, and both matter: the gradient norm will never exceed θ, and the updated gradient g′ remains entirely aligned with the original direction of g. Clipping changes the step length, never the step direction.

11. Chapter summary

• Backpropagation is the most fundamental component of an ANN.
• It aims to minimize the loss function by adjusting the network weights and biases.
• The level of adjustment is determined by the gradient of the loss function with respect to those parameters.
• That gradient can be computed, for each weight, by the chain rule.
• Backpropagation can be implemented in matrix form to take advantage of fast matrix libraries and powerful GPU-based computing systems.
• Vanishing and exploding gradients are common issues in deep neural networks. ReLU, careful weight initialization, gradient clipping and batch normalization help to solve or at least mitigate these problems.

Check your understanding