The deck opens by redrawing the pipeline of Chapter 5 and pointing at the last box: we now focus on the model, the "heart" of the AI in our system.
The lecturer also lists the interactive demos worth playing with: Playground TensorFlow, Convnetjs, LLM architectures, Transformers, and an interactive gradient descent visualisation. And a disclaimer: neural networks are mathematically intensive.
Neural Networks (NN): the key idea is to imitate, as far as possible, the neurons of the human brain — networks because neurons are connected to each other. The biological picture the deck starts from:
The first scheme about the neuron was introduced by McCulloch and Pitts (1943); the first Artificial Neuron (AN) was introduced by Rosenblatt in 1957. Its four design decisions:
Will I pass the Machine Learning exam?
y' = 1 if 0.3 * x1 + 0.8 * x2 + 0.5 * x3 >= 3
0 otherwiseThe deck asks how to implement not, and, or with a single artificial neuron — all three are shown to work — and then asks the decisive question: what about xor?
| x1 / x2 | 0 | 1 | x1 / x2 | 0 | 1 | |
|---|---|---|---|---|---|---|
| and 0 | 0 | 0 | or 0 | 0 | 1 | |
| and 1 | 0 | 1 | or 1 | 1 | 1 | |
| xor 0 | 0 | 1 | not: x1 = 0 → 1, x1 = 1 → 0 | |||
| xor 1 | 1 | 0 | ||||
The answer, stated flatly on the slide: a single AN can solve only linear problems. The solution is to use more ANs organized on different layers → the Multi-Layer Perceptron (MLP). It is not so easy, since this introduces several mathematical problems, and it greatly increases the computational load.
Set the two weights and the threshold, and try to reproduce each Boolean function. XOR will resist.
Neural networks are groups of artificial neurons organized in different layers:
Each neuron is fully connected to those of the next level. Again, this imitates the hierarchical nature of our neurons: we have "only" about ten levels between the retina and the actuator muscles — otherwise we would be too slow to react to stimuli.
| Topology | Definition | Suitable for |
|---|---|---|
| Feed forward | The connections connect the neurons of one level with the neurons of the next level. Backward connections or connections to the same level are not allowed. | The only kind used in this course |
| Recurrent | Feedback connections are expected — generally towards neurons of the same level, but also backward. | Sequences, because they have a (short-term) memory effect |
Activation functions define the output of the neuron given an input or set of inputs:
The three shown in the deck are ReLU, Sigmoid and Linear.
A NN with at least 1 hidden layer can approximate any continuous function to any desired degree of accuracy, given sufficient neurons in that hidden layer, the right weights and biases, and a non-polynomial activation function.
And immediately, its four limitations — which are the interesting half:
Håstad's switching lemma (Hastad 1986): certain functions, while easily represented by deep networks with a modest number of neurons, require an exponentially larger number of neurons to represent accurately if constrained to a single hidden layer. "Global" functions are harder to approximate, for instance:
This lemma provides a theoretical foundation for using depth in both circuit complexity and neural networks: it justifies the need for a layered structure when working with complex, globally-dependent functions.
Kidger and Lyons (2020): let n be the number of input neurons, m the number of output neurons, and let ρ be any nonaffine continuous function with a continuous, nonzero derivative at some point. The class of neural networks of arbitrary depth, width n + m + 2, and activation function ρ, is dense in C(K; Rm) for K ⊆ Rn compact. This covers any activation function, including polynomial ones.
The deck draws the conclusion in one line: this is why deep neural networks work.
Read the three results as one argument. The classical theorem says width is enough in principle; Håstad says width can cost exponentially more than depth; Kidger and Lyons say depth is enough too, with a fixed narrow width. Together they explain the shape of modern networks — deep rather than merely wide — and, importantly, none of the three tells you how to find the weights. That question is answered by gradient descent and backpropagation, further down this page.
General considerations about neural network architectures: a greater number of hidden layers (therefore neurons) means better performance, but also the need for more training data and a greater computational load. Training a neural network is complicated, but specific frameworks exist — TensorFlow and PyTorch — exactly as scikit-learn plays that role in classical machine learning.
Deep Learning is a branch of ML that avoids the problematic phase of feature extraction (also) with high-dimensional inputs. Feature extraction requires human intervention — and often, this is the weak link in the chain.
Training a NN means tuning the weights to optimize the prediction accuracy, by minimizing a loss/cost function. The visual guide given on the slide, point by point:
The cost function is a mathematical formulation of the learning goal: it measures the error between the prediction and the ground truth (label), and presents the performance (or error) in the form of a single real number.
Cross Entropy is the distance between what the model believes the output distribution should be and what the original distribution is. To use the Cross Entropy loss, the output layer must output probabilities.
Binary Cross Entropy (BCE)
Loss = -yi * log(y_hat_i) - (1 - yi) * log(1 - y_hat_i)
Categorical Cross Entropy (CCE)
Loss = - SUM over i of yi * log(y_hat_i)y_hat_i is the i-th scalar value in the model output (the prediction), yi is the corresponding target (label) value;-yi * log(y_hat_i) cancels out if the target is 0;(1 - yi) * log(1 - y_hat_i) cancels out if the target is 1.The softmax layer transforms an n-dimensional vector of real numbers into a vector of real numbers in [0, 1] which adds up to 1. The softmax activation function determines the final probability value of each class:
p_i = e^(a_i) / SUM over k = 1..n of e^(a_k)Softmax is a continuously differentiable function — which is not a detail, because the whole training procedure requires differentiability.
The deck follows one image through training: at first the network assigns a moderate probability to the correct class and the loss is 0.3677; after some training iterations the probability of the correct class is higher and the loss is 0.0923. The loss function has decreased.
The text extracted from the slide prints CCE = −log2 0.755 = 0.3677 and CCE = −log2 0.0938 = 0.0923. Neither line is arithmetically consistent as written, but the intended values are recoverable: 2^(−0.3677) = 0.775 and 2^(−0.0923) = 0.938. So the story is: the probability assigned to the correct class rises from about 0.775 to about 0.938, and the loss correspondingly falls from 0.3677 to 0.0923. Use the widget below to check the relationship yourself — that is what the slide is teaching, and it holds regardless of the typography.
How to minimize the loss? Adjusting (changing) the weights and the bias of every neuron.
Gradient descent is a method for unconstrained mathematical optimization: a first-order iterative algorithm for minimizing a differentiable multivariate function F(x). It takes repeated steps in the opposite direction of the gradient of F(x) at the current point — the direction of steepest descent. Conversely, stepping in the direction of the gradient leads to a trajectory that maximizes the function (gradient ascent).
If F(x) is defined and differentiable in a neighborhood of a point a, then F(x) decreases fastest in the direction of the negative gradient of F at a, that is −∇F(a). It follows that if
a(n+1) = a(n) - eta * grad F( a(n) )then for a small enough step size or learning rate η ∈ R+, we have F(an) ≥ F(an+1).
grad J(Theta).Theta = Theta - eta * grad J(Theta).Smooth convergence, but not recommended for huge training datasets: it is a slow and computationally expensive algorithm.
Theta = Theta - eta * grad J(Theta; B).Advantages: faster training — more frequent parameter updates than batch gradient descent, leading to faster convergence; efficient GPU/CPU utilization — mini-batches enable parallel processing.
Drawbacks: selecting an optimal batch size requires experimentation, as it affects both the speed and the stability of convergence; mini-batch gradient descent is less stable than full-batch gradient descent, especially for small batch sizes.
Adam (Adaptive Moment Estimation) is a stochastic gradient descent method with fewer parameters than SGD. It is based on two principles:
It is the default solver of scikit-learn's MLPClassifier.
Set the learning rate and step. Too small and you crawl; too large and you overshoot the minimum entirely.
| Term | Definition |
|---|---|
| Batch (size) | The hyperparameter of gradient descent that controls the number of training samples to work through before the model's internal parameters are updated |
| #Iterations | The size of the training set divided by the batch size |
| Epoch | The learning procedure is applied to the entire training dataset: one epoch means that each sample in the training set has had an opportunity to update the weights |
The practical notes on batch size:
And on epochs: an epoch is comprised of one or more batches (or iterations), and the number of epochs is traditionally large (10, 100, 1000, …).
We can apply the Widrow-Hoff (delta) rule on the perceptron — a simple gradient descent technique. For a neuron j with activation function g(x), the delta rule for neuron j's i-th weight wji is:
delta w_ij = eta * (yi - y_hat_i) * g'(h_j) * xixi is the i-th input;h is the weighted sum of inputs, h_j = SUM over i of xi * w_ji;g'() is the derivative of the activation function;y_hat_i is the actual output, yi is the target output;eta is the learning rate.Backpropagation — backward propagation of errors — propagates the error to the input of an ANN.
The introduction of backpropagation has been fundamental in training DNNs, and the delta rule is a special case of the more general backpropagation algorithm.
The four steps:
y_hat = f^L(W^L f^(L-1)(W^(L-1) ... f^1(W^1 x) ...)).(x, y) it is C(y, f^L(W^L f^(L-1)(W^(L-1) ... f^2(W^2 f^1(W^1 x)) ...))).The chain rule is used to differentiate composite functions: given f() and g() with y = f(g(x)), then dy/dx = dy/dg * dg/dx. The worked example on the slide:
y = sin(x^2)
dy/dg = cos(g) ; dg/dx = 2x ; finally dy/dx = cos(x^2) * 2xIn a DNN the same rule is chained through every hidden layer — where x is the input, zl is the weighted input of hidden layer l and al is its output — producing the long product of partial derivatives from the loss all the way back to the input.
And the practical warning attached to it: the choice of the right value of the learning rate is important.
It is common to create line plots that show epochs along the x-axis and the loss value of the model on the y-axis. These plots are called learning curves:
Every argument in that constructor is a concept from this chapter: the architecture, the activation function, the optimizer, the batch size, the learning rate schedule, the number of epochs, and the shuffling that mini-batch SGD requires.
It is not just important how well a particular classifier works. The main steps of an AI project are known — data collection, data processing (and feature extraction), model training, prediction analysis through metrics — but other considerations remain: when to address a problem through ML? when through DL? what hardware and software resources are needed?
| Element | Machine Learning | Deep Learning |
|---|---|---|
| Data | Large data (~ hundreds) | Huge data (~ thousands) |
| Accuracy | High accuracy | Best accuracy (high-dimensional data) |
| Training time | ~ minutes | ~ hours, days |
| Hardware | CPU | GPU |
| Features | Manual | Learned |
| Interpretability | Good | Low |
The training of NNs, especially deep ones, requires specialized hardware: before starting a project with DL, you need to ask if the company or lab has the necessary hardware. Having one or more GPUs is today a fundamental factor — GPUs are essential for parallelizing (and therefore speeding up) calculations, and the deeper a network is, the more computational load is introduced. To date Nvidia dominates the market: parallelization is made possible by the CUDA libraries (Compute Unified Device Architecture, the true core business of Nvidia), and Google has started a competition by introducing the TPU (Tensor Processor Unit).
| In-house solution | External solution (PaaS / cloud) | |
|---|---|---|
| What it is | The company buys the necessary hardware and is the direct owner | The hardware is rented through the PaaS paradigm |
| Pros | Extreme freedom of use of hardware; in the long run it tends to have lower costs | Hardware maintenance is not required; no investment over time for upgrades; dedicated server rooms are not required and energy consumption is not borne by the company |
| Cons | Hardware maintenance requires specialized technicians; hardware ages quickly; for large numbers of GPUs you need server rooms with temperature and access control, and high energy consumption (a latest-generation GPU draws ~450 W); the GPU market is expensive and volatile — very few companies are involved in production (TSMC), semiconductor shortages, and external factors such as mining and wars | In the long run it tends to have higher costs; plus the classic cloud problems: vendor lock-in, who really owns the data, and privacy issues |
The deck points at cost calculators (Google Cloud, LeaderGPU, Colab Pro, AWS) and closes the economic argument with two references: the piece "Saying 'Thank You' to ChatGPT Is Costly. But Maybe It's Worth the Price", and Cottier et al. (2024) on the rising costs of training frontier AI models. The very last slide returns to where Chapter 1 started: classic programming on one side, deep learning on the other.
The first scheme of the neuron is by McCulloch and Pitts (1943); the first Artificial Neuron by Rosenblatt in 1957. Its inputs are digital numbers (not analog signals); they are weighted, because signals are not all equally important; they are merged with a sum function plus a bias; and an activation function generates the final output — like the brain, which filters inputs because it is impossible to always take everything into account. The example given: y' = 1 if 0.3·x1 + 0.8·x2 + 0.5·x3 ≥ 3, else 0.
Because a single AN can solve only linear problems: it computes a weighted sum and thresholds it, which draws a single straight boundary in the input space. AND and OR are linearly separable, XOR is not — no single line puts the two 1-outputs on one side. The solution is to use more ANs organized on different layers, the Multi-Layer Perceptron; this is not easy, since it introduces several mathematical problems and greatly increases the computational load.
An input layer, an output layer and one or more hidden layers, with each neuron fully connected to those of the next level. Feed forward: connections go from one level to the next, and backward connections or connections to the same level are not allowed. Recurrent: feedback connections are expected, generally towards neurons of the same level but also backward, which gives a short-term memory effect and makes them suitable for sequences. This course uses only feed-forward networks.
It defines the output of the neuron given an input or set of inputs: it outputs a small value for small inputs and a larger value if the inputs exceed a threshold. It is a sort of switch of the artificial neuron. The three shown are ReLU, Sigmoid and Linear.
A NN with at least 1 hidden layer can approximate any continuous function to any desired degree of accuracy, given sufficient neurons in that hidden layer, the right weights and biases, and a non-polynomial activation function. Limitations: it applies to feedforward networks with n inputs, a single hidden layer and 1 output; a close approximation may need an impractically large number of neurons, making the network hard to train; it assumes the right weights and biases exist but does not say how to find them; and it is not applicable to discontinuous functions.
It says that certain functions, easily represented by deep networks with a modest number of neurons, require an exponentially larger number of neurons if constrained to a single hidden layer; "global" functions are harder to approximate. The parity function determines whether the number of 1s in a binary input string is odd or even; the majority function outputs 1 if more than half of the input bits are 1. The lemma is the theoretical foundation for using depth, in circuit complexity and in neural networks alike.
A branch of ML that avoids the problematic phase of feature extraction, also with high-dimensional inputs. Feature extraction requires human intervention, and this is often the weak link in the chain. The trade-off appears in the comparison table: more data (~thousands vs ~hundreds), best accuracy on high-dimensional data, training in hours or days rather than minutes, GPU rather than CPU, features learned rather than manual, and low interpretability instead of good.
Cross entropy is the distance between what the model believes the output distribution should be and what the original distribution is; to use it, the output layer must output probabilities. BCE: Loss = −yi·log(y_hat_i) − (1 − yi)·log(1 − y_hat_i), where the first term cancels out if the target is 0 and the second if the target is 1. CCE: Loss = − SUM_i yi·log(y_hat_i).
It transforms an n-dimensional vector of real numbers into a vector of real numbers in [0, 1] which adds up to 1, determining the final probability value of each class: p_i = e^(a_i) / SUM_k e^(a_k). It is continuously differentiable, which matters because the whole training procedure requires differentiability.
a(n+1) = a(n) − eta · grad F(a(n)). Gradient descent is a first-order iterative algorithm for minimizing a differentiable multivariate function, taking repeated steps in the opposite direction of the gradient — the direction of steepest descent. For a small enough step size or learning rate eta > 0, F(an) ≥ F(an+1): each step is proportional to the learning rate, and choosing it well is important — too small and convergence crawls, too large and the steps overshoot.
Vanilla: compute the cost and gradient over the entire dataset, then update once per epoch — smooth convergence, but slow and computationally expensive, not recommended for huge datasets. Mini-batch SGD: split into small batches and update once per batch, shuffling after each epoch — faster training and efficient GPU/CPU utilization, at the cost of needing experimentation for the batch size and being less stable, especially for small batches. Adam: an SGD method with fewer parameters, combining momentum (an exponentially decaying average of past gradients, smoothing them) and adaptive learning rates (a moving average of squared gradients; parameters with large gradient changes get smaller learning rates and vice versa).
Batch size: the hyperparameter controlling the number of training samples to work through before the model's internal parameters are updated; typically a multiple of a power of two (16, 32, 64), too small compromises stability, larger uses more video memory, and empirically it may be at least equal to the number of classes. #Iterations = size of the training set / batch size. Epoch: the learning procedure applied to the entire training dataset — each sample has had an opportunity to update the weights; an epoch comprises one or more batches, and the number of epochs is traditionally large (10, 100, 1000).
delta w_ij = eta (yi − y_hat_i) g'(h_j) xi, the Widrow-Hoff rule, a simple gradient descent technique for the perceptron. xi is the i-th input; h_j = SUM_i xi·w_ji is the weighted sum of inputs; g'() is the derivative of the activation function; y_hat_i is the actual output and yi the target; eta is the learning rate. The delta rule is a special case of backpropagation.
(1) Forward pass: compute outputs layer by layer up to a prediction. (2) Loss calculation: quantify the difference between prediction and true output. (3) Backward pass: compute gradients by propagating the error backward to each layer, using the chain rule. (4) Parameter update: adjust each weight and bias by gradient descent. The chain rule is needed because we cannot directly compute the derivative of the loss with respect to the network outputs, and because backpropagation is a local process — neurons are completely unaware of the complete topology. Example: y = sin(x²) gives dy/dx = cos(x²)·2x.
Line plots with epochs on the x-axis and the loss value on the y-axis, usually two overlapped — one for training, one for validation. Since these are loss values, low is good. They help diagnose whether the model has over-learned (overfitting), under-learned (underfitting), or suitably fits the training set. Typically the loss drops rapidly at the beginning; the point where the validation curve stops improving while the training curve keeps falling marks the start of overfitting.
In-house: extreme freedom of use and lower costs in the long run, but hardware maintenance requiring specialized technicians, quick ageing and repeated investment, and — with many GPUs — server rooms with temperature and access control plus high energy consumption (a latest-generation GPU draws about 450 W); the GPU market is expensive and volatile, with very few producers (TSMC), semiconductor shortages, and external factors such as mining and wars. External / PaaS: no maintenance, no upgrade investment, no server room or energy bill, but higher costs in the long run plus the classic cloud issues — vendor lock-in, data ownership, and privacy.