Linear algebra, calculus and automatic differentiation

1. The basic skills we need

Before any network can be trained, three mathematical tools have to be in place. The slides state the reason for each one, and each reason maps to a later chapter of this course.

2. Vectors, matrices and tensors

Three objects, one idea: an array with a growing number of axes.

Tensors become important when working with color images, which arrive as arrays with 3 axes: height, width, and a channel axis for stacking the color channels (RGB). Indexing works like matrices: given an RGB image X, the element xijk is the value at the i-th row, j-th column and k-th channel.

3. Operations on tensors

Transpose and trace

The transpose flips a matrix over its diagonal by exchanging rows and columns: given A ∈ ℝm×n, its transpose AT is an n×m matrix. The trace of a square matrix A ∈ ℝn×n is the sum of the elements on its main diagonal:

tr(A) = ∑i=1n aii

Element-wise and scalar operations

Given any two tensors of identical shape, the result of any binary element-wise operation is a tensor of the same shape. For two matrices A, B ∈ ℝm×n:

• Addition: each entry is aij + bij.
• Element-wise multiplication (the Hadamard product, written A ⊙ B): each entry is aij · bij.

Adding or multiplying a tensor by a scalar does not change its shape: every element is added to, or multiplied by, the scalar. Likewise, applying a real function f: ℝ → ℝ (for instance sin) to a tensor yields a tensor of the same shape with f applied element by element. This is exactly how activation functions are applied to a whole layer in Chapter 3.

Average of a tensor

The average μ(A) is a scalar: the sum of all elements divided by the total number of elements. For A ∈ ℝm×n that denominator is m · n. This tiny definition returns as the μ in the Pix2Pix and CycleGAN loss functions of Chapter 11.

Dot product, matrix product, Frobenius inner product

The matrix product has one hard constraint: the number of columns in the first matrix must equal the number of rows in the second. The result has the rows of the first and the columns of the second. Try it below: this shape rule is the single most common source of runtime errors when you build a network by hand.

4. Norms

A norm maps a vector to a scalar that tells us how big the vector magnitude is. It is one of the most useful operators in linear algebra, and in this course it appears as a regularization penalty (Chapter 9), as a gradient clipping threshold (Chapter 4) and as a loss function (Chapter 3).

A vector norm f: ℝn → ℝ must satisfy four properties:

• f(αx) = |α| · f(x) — absolute homogeneity;
• f(x + y) ≤ f(x) + f(y) — the triangle inequality;
• f(x) ≥ 0;
• f(x) = 0 ⇔ xᵢ = 0 for all i.

The L_p norm of a vector x ∈ ℝn is:

‖x‖p = ( ∑i=1n |xi|p )1/p

The Euclidean distance is the L₂ norm: ‖x‖2 = √(∑i=1n xi2). The subscript is often omitted, so a bare ‖x‖ means the L₂ norm.

The Frobenius norm of a matrix A ∈ ℝm×n is the square root of the sum of the squares of its elements:

‖A‖F = √( ∑i ∑j aij2 )

The Frobenius norm satisfies all the properties of vector norms — it is, in effect, the L₂ norm of the matrix flattened into a vector.

5. Derivatives and differentiation rules

In DL we use loss functions that are differentiable with respect to the model parameters. This means that for each parameter we can determine how rapidly the loss would increase or decrease, were we to increase or decrease that parameter by an infinitesimally small amount. That single sentence is the whole reason calculus is in this course.

Given f: ℝ → ℝ, y = f(x), the derivative can be written in several equivalent ways:

f′(x) = y′ = ∂y/∂x = ∂f/∂x = (∂/∂x) f(x)

6. Partial derivatives, gradient and Jacobian

In DL, functions often depend on many variables. To calculate the partial derivative of f(x₁, …, xₙ) with respect to xᵢ, we treat the other variables as constants and differentiate with respect to xᵢ. It is written ∂y/∂xᵢ, ∂f/∂xᵢ or f′xᵢ.

Gradient

The partial derivatives of a multivariate function with respect to all its variables are concatenated into the gradient vector. Given f: ℝn → ℝ and x ∈ ℝn, the gradient ∇f(x) is a vector of n partial derivatives:

∇f(x) = [ ∂f/∂x₁,  ∂f/∂x₂,  …,  ∂f/∂xn ]T

Jacobian matrix

The Jacobian generalises the gradient to a multivariate vector function. Given f: ℝn → ℝm taking x ∈ ℝn and producing y = f(x) ∈ ℝm, the Jacobian Jf (also written ∇f) is an m×n matrix whose i-th row is ∇fiT, the transpose of the gradient of the i-th component:

Jf = [ ∂fi/∂xj ]   with  1 ≤ i ≤ m,  1 ≤ j ≤ n

7. Interpreting the gradient

The slides work an example that is worth reproducing in full, because the intuition it builds is the one that makes gradient descent obvious later.

Consider the multiplication of two numbers, f(x, y) = x · y. By the constant multiple rule:

• ∂f/∂x = 1 · y = y
• ∂f/∂y = x · 1 = x

so the gradient vector is ∇f = [y, x]T. Now instantiate it at x = 4, y = −3, where f(x, y) = −12:

The slides read this as a sensitivity: the partial derivative of f with respect to each variable is the sensitivity of the function in the surrounding of its current value. From that reading the operating rule follows directly.

Turn it upside down — we want the loss lower, not higher — and you have the gradient descent update of Chapter 5: move each parameter against its partial derivative.

8. Composite functions and the chain rule

Consider the composite function f(x, y, z) = (x + y) · z. It decomposes into two primitives:

• q(x, y) = x + y
• f(q, z) = q · z

The partial derivatives of both are immediate:

But we are not interested in the intermediate value ∂f/∂q. What we want are the partial derivatives of f with respect to its inputs x, y and z. The chain rule delivers them:

∂f/∂x = (∂f/∂q) · (∂q/∂x) = z · 1 = z
∂f/∂y = (∂f/∂q) · (∂q/∂y) = z · 1 = z

Notice the structure: a local gradient (∂q/∂x, known at the node) multiplied by an upstream gradient (∂f/∂q, arriving from above). Chapter 4 gives this pattern a name and applies it to every operator in a neural network.

9. Automatic differentiation

Differentiation is a crucial step in nearly all DL optimization algorithms. Automatic differentiation is a set of techniques to numerically evaluate the derivative of a function specified by a computer program, and derivatives of arbitrary order can be computed automatically by applying the chain rule.

In reverse mode — the mode used by every deep learning framework — derivatives are computed in two phases:

1. Forward phase: the original function is decomposed into a set of primitive functions using a computational graph, then each of these expressions is evaluated.
2. Reverse phase: derivatives are calculated by applying the chain rule from the outputs to the inputs.

The worked example

Take the same function, f(x, y, z) = (x + y) · z, at the point (x, y, z) = (−2, 5, −4). The slides walk it operator by operator.

Step through it yourself. Advance the forward pass to fill the node values, then the reverse pass to propagate the gradients back to the inputs.

The same thing in code

The laboratory of this course uses TensorFlow with Keras. Its automatic differentiation entry point is tf.GradientTape, which records the forward phase and replays it in reverse on demand.

The equivalent in PyTorch, the other framework discussed in the guest lecture, uses the tensor graph directly:

import torch

x = torch.tensor(-2.0, requires_grad=True)
y = torch.tensor( 5.0, requires_grad=True)
z = torch.tensor(-4.0, requires_grad=True)

q = x + y          # forward phase: primitive 1
f = q * z          # forward phase: primitive 2
f.backward()       # reverse phase: chain rule, outputs to inputs

print(f.item())    # -12.0
print(x.grad, y.grad, z.grad)   # tensor(-4.) tensor(-4.) tensor(3.)

10. Deep learning frameworks

A guest lecture by Lorenzo Pellegrini (RTDa, University of Bologna) surveys the tools that implement everything above. It starts from the machine learning software stack, where the general-purpose landscape splits neatly into two:

• for prototyping and R&D — Scikit-learn (Python), Caret (R), Weka (Java);
• for production — Kubernetes, KubeFlow, Hadoop; H2O, MLlib, SparkML, FlinkML.

For deep learning, the lecture observes, the line between prototyping and production is more blurred: Caffe / Caffe 2, Torch / PyTorch, TensorFlow 2, plus CNTK, MxNet, Gluon and Chainer. The reason is a narrower focus:

1. (Deep) neural networks;
2. gradient-based optimization;
3. transparent (multi) CPU/GPU acceleration.

Prototyping with neural nets on massive datasets needs efficiency, so the same engine has to serve both roles.

The seven axes of comparison

The lecture proposes seven features along which open-source DL frameworks differ. They are a useful checklist when you choose a stack for the team project.

TensorFlow and PyTorch side by side

Their distinctive strengths, as the lecture lists them:

• TensorFlow — great models and utils support; large API variants at different levels, but Keras is now the mainly used one; solid codebase; amazing community support and a permissive Apache 2.0 licence.
• PyTorch — great models and utils support; nice integration with ONNX and Caffe2; great for highly-dynamic graphs; simple, neat API; easy to learn.

11. Chapter summary

• Scalars, vectors, matrices and tensors are the basic mathematical objects in linear algebra, and we can perform a variety of operations over them.
• In DL we often work with norms such as the L₁, L₂ and Frobenius norms.
• A derivative can be interpreted as the instantaneous rate of change of a function with respect to its variable, or as the slope of the tangent line to the curve.
• A gradient is a vector whose components are the partial derivatives of a multivariate function with respect to all its variables; the Jacobian contains all the partial derivatives of a multivariate vector function.
• Composite functions are differentiated by applying the chain rule.
• DL frameworks can automate the calculation of derivatives using automatic differentiation.

Check your understanding