The chapter opens with the guiding metaphor: “Artificial neural networks paradigm is inspired by the way the biological nervous system processes information. It is composed of large number of highly interconnected processing elements working in unison to solve a specific problem.”
Biological neurons are the fundamental units of the brain and nervous system, and the slides break one into four parts. Each has an exact counterpart in the artificial model.
| Biological part | Function | Artificial counterpart |
|---|---|---|
| Dendrites | Collect incoming signals (inputs) | The inputs in1 … ind |
| Soma | Processes the incoming signals over time and converts the processed value into an output | The weighted sum net followed by the activation function f |
| Axon | Works as a transmission line | The output value out |
| Synapses | At the end of the axon, connect to other neurons to transmit the output signal | The weighted edges to the next layer |
The scale argument follows. A single biological neuron is a weak element, but connected with billions of other neurons it becomes a powerful network called the brain. The human brain contains about 100 billion (1011) neurons that communicate by electric and chemical signals through more than 100 trillion (1014) synapses.
The unit is deliberately trivial: a weighted sum and a non-linear squashing. All the expressive power comes from connecting many of them, and all the difficulty comes from finding the weights. Chapter 4 solves the second problem.
The artificial neuron model was first proposed by McCulloch and Pitts in 1943 and designed to mimic the behaviour of biological neurons. It receives one or more inputs and sums them to produce an output (or activation). Each input is separately weighted, and the sum is passed through a non-linear function called the activation function.
Formally, in two steps:
net = w₀ + ∑i=1d ini · wi
out = f(net)| Symbol | Meaning |
|---|---|
in1, …, ind | The neuron inputs. |
w1, …, wd | Weights assigned to each input. |
w₀ (the bias) | Allows the activation function to be shifted by adding a constant to the input. The bias is used to delay the triggering of the activation function. It is drawn as a virtual input in₀ = 1. |
net | The weighted sum of all the inputs, computed as the first step. |
f | The activation function. |
out | The output, or activation, of the neuron. |
w₀, which is why it shifts the activation function rather than scaling anything.Drive one yourself. Move the weights and the inputs and watch net and out respond; switch the activation function to see how the same net maps to very different outputs.
Similar to the brain, an Artificial Neural Network (ANN) is made up of artificial neurons connected to each other. The slides establish the vocabulary:
Two named architectures follow from this:
| Name | Definition |
|---|---|
| Feed-Forward Neural Network (FFNN) | An ANN where connections between neurons do not form a cycle. |
| Multi-Layer Perceptron (MLP) | The most common FFNN, consisting of three or more layers: an input layer, one or more hidden layers, and an output layer. MLPs are fully-connected: each neuron in one layer is connected with every neuron in the following layer. |
The learning process is the key feature of ANNs and, the slides note, it is closely related to how the human brain learns. Iteratively:
Two clarifications the slides add:
The worked example on the slides is a classifier that must separate cats from dogs. An image enters the input layer, propagates through the hidden layer to the output layer, and the error is computed as:
Error = predicted output − desired outputand that error drives the weights adjustment.
The goal of a good machine learning model is to generalize well from the training data to any data from the problem domain. This allows predictions to be made on data the model has never seen. Two failure modes stand between us and that goal.
| Overfitting | Underfitting | |
|---|---|---|
| Definition | The phenomenon of fitting the training data more closely than the underlying distribution. | A network that can neither model the training data nor generalize to new data. |
| Why it happens | The danger when working with finite training samples: we discover apparent associations that are not present in the underlying population from which our training set was drawn. | The model or the training is not sufficient for the task. |
| How to detect it | The model performs very well on the training set but relatively poorly on the validation set. | The model performs poorly on the training set. |
| Solutions | Increase the size of the training set; reduce the complexity of the network. | Increase the complexity or the type of the model; increase the training time to minimize the cost function. |
The diagnosis rests on watching two curves during training: the training error and the generalization (or validation) error. Plotted against training time or model complexity, the training error keeps falling while the generalization error eventually turns and rises. The gap between the two curves is the overfitting.
The diagnostic rule is asymmetric and is asked often. Underfitting is diagnosed on the training set alone (the model is bad even there). Overfitting requires both sets (very good on training, relatively poor on validation). Quoting only the validation error cannot distinguish the two.
One of the two cures for overfitting is “increase the size of the training set” — but obtaining new training data is not always a feasible option. The training set can instead be increased by generating new instances of the same data with some transformations.
Data augmentation identifies a set of techniques used to increase the quantity, quality and variability of the training set by adding modified copies of existing data or by generating new synthetic data. The techniques the slides list:
| Geometric | Photometric |
|---|---|
| flipping · translation · rotation · scaling · zooming · stretching · cropping | noise · change in brightness and contrast |
The two-column split is an editorial grouping for memorability; the slides give one flat list. The distinction is useful anyway: geometric augmentations teach invariance to where and how big the object is, photometric ones teach invariance to capture conditions. AlexNet in Chapter 7 uses data augmentation for exactly this reason.
The activation function decides whether or not a neuron should be activated, by calculating the weighted sum and adding the bias to it. But the deeper justification is this:
Without an activation function, the output signal would be a simple linear function, and the neuron could not learn and model complicated data such as images, audio or speech. The activation function introduces non-linearity into the network, allowing it to learn and compute more complex functions. Activation functions are also needed to restrict the output value into a certain finite range.
Several activation functions have been proposed on the basis of five properties. Learn these five: every entry in the catalogue below is scored against them.
| Property | Why it matters |
|---|---|
| Nonlinear | If the activation function is non-linear, then a two-layer ANN can be proven to be a universal function approximator. |
| Range | If the output range is finite, training methods tend to be more stable; otherwise training is more efficient but a smaller learning rate is necessary. |
| Continuously differentiable | Desirable for enabling gradient-based optimization methods. |
| Monotonic | If the activation function is monotonic, the error surface associated with a single-layer model is guaranteed to be convex. |
| Approximates identity near the origin | With this property the ANN learns efficiently when its weights are initialized with small random values; otherwise special care must be used when initializing the weights. |
Eight functions, each with its formula and its trade-off. Explore them in the plotter, then read the trade-offs in the tabs below it.
A threshold-based activation function. Only if the input value is above a certain threshold is the neuron activated and sends its signal to the next layer.
f(x) = 0 if x < 0
f(x) = 1 otherwise| Advantages | Disadvantages |
|---|---|
| The output range is finite: [0, 1]. It is monotonic. |
It does not allow multi-value outputs. It is not differentiable at 0. It does not approximate identity near the origin. |
The identity function creates an output signal equal to the input: f(x) = x. It is the activation used in the worked backpropagation example of Chapter 4, precisely because its derivative is 1 and disappears from the arithmetic.
| Advantages | Disadvantages |
|---|---|
| It allows multiple outputs. It is monotonic. It is continuously differentiable. It approximates identity near the origin. |
It is not non-linear. The output range is not finite. Its derivative is a constant. |
The sigmoid transforms its inputs to outputs that lie on the interval (0, 1):
σ(x) = 1 / (1 + e−x)| Advantages | Disadvantages |
|---|---|
| Can be used for models where we need to predict a probability as an output. Smooth gradient, preventing jumps in output values. Output values bound between 0 and 1, normalizing the output of each neuron. It is monotonic. It is continuously differentiable. |
It does not approximate identity near the origin. For very high or very low inputs there is no change to the prediction, causing the vanishing gradient problem. Outputs are not zero centred. Computationally expensive. |
Its derivative, which Chapter 4 uses to collapse five operators into one, is σ′(x) = (1 − σ(x)) · σ(x).
The hyperbolic tangent maps inputs into the interval (−1, 1):
tanh(x) = (1 − e−2x) / (1 + e−2x)| Advantages | Disadvantages |
|---|---|
| Zero centred, making it easier to model inputs that have strongly negative, neutral and strongly positive values. Smooth gradient. Output bound between −1 and 1, normalizing the output of each neuron. It is monotonic. It is continuously differentiable. It approximates identity near the origin. |
For very high or very low inputs there is no change to the prediction, causing the vanishing gradient problem. Computationally expensive. |
tanh is the activation used in LeNet-5 (Chapter 7) and inside the LSTM cell (Chapter 8).
Given an element x, the Rectified Linear Unit is the maximum of that element and 0:
ReLU(x) = max(x, 0)| Advantages | Disadvantages |
|---|---|
| It is monotonic. Computationally very efficient. |
The output range is not finite. It is not continuously differentiable. When inputs approach zero or are negative, the gradient of the function becomes zero and the network cannot learn (dying / dead neurons). |
Despite the dead neuron problem, ReLU is the simplest solution to the vanishing gradient problem, because it saturates in only one direction (Chapter 4).
The Leaky Rectified Linear Unit gives the negative branch a small slope:
LeakyReLU(x) = max(0.01 · x, x)| Advantages | Disadvantages |
|---|---|
| It is monotonic. It enables backpropagation even for negative input values, addressing the problem of dying / dead neurons. |
The output range is not finite. It is not continuously differentiable. It does not provide consistent predictions for negative input values. |
The Exponential Linear Unit replaces the leaky straight line with an exponential curve:
ELU(x) = x if x > 0
ELU(x) = α · (ex − 1) otherwisewhere the hyper-parameter α > 0 is usually set to 1.
| Advantages | Disadvantages |
|---|---|
| It is monotonic. It enables backpropagation even for negative input values, addressing the dying / dead neuron problem. |
The output range is not finite. It is not continuously differentiable. Slow convergence due to the exponential function. |
Softmax is different in kind from the seven above: it does not act on one number but on a whole layer. It not only maps the output to a (0, 1) range but also maps each output of a layer in such a way that the total sum is 1:
softmax(x)i = exi / ∑j=1n exj, x ∈ ℝnBecause the output of softmax is a probability distribution, it is typically used in the output layer to classify inputs into multiple categories. The slides give a worked instance: the output layer values (1.3, 5.1, 2.2, 0.7, 1.1) become the probabilities (0.02, 0.90, 0.05, 0.01, 0.02).
To improve the performance of neural network models, the error of the current state of the model must be repeatedly estimated. This requires choosing a function, conventionally called the loss function, to estimate the error of the model so that it can be updated to reduce the error on the next evaluation. In other words, the loss function measures how good a neural network model is at predicting the expected outcome.
Although loss and cost are usually treated as synonyms, the slides insist they are different: a loss function is for a single training example; a cost function is the average loss over the entire training dataset. The optimization strategies aim at minimizing the cost function.
Generally, the cost function is computed as:
C = ( ∑i=1n L(yi, ŷi) ) / nwhere n is the number of training examples, yi is the actual observation of the i-th training example, ŷi is its prediction, and L is the loss function.
There is not a single loss function that works for all kinds of problems and data. The choice depends on several factors such as the presence of outliers in the data, time efficiency, and ease of finding the gradient. Loss functions split into two families according to the learning task: regression losses and classification losses.
Regression predictive modeling is the task of approximating a mapping function from input variables to a continuous output variable, such as an integer or floating-point value. The three most common regression losses:
| Loss | Formula | Cost function | Behaviour |
|---|---|---|---|
| Square error (quadratic, L2 loss) |
L = (y − ŷ)2 |
MSE, the average of squared differences between predictions and actual observations | The most used regression loss. The result is always positive regardless of the signs, and a perfect value is 0. The squaring means predictions far from the actual values are penalized heavily compared with less deviated predictions. |
| Absolute error (L1 loss) |
L = |y − ŷ| |
MAE, the average of absolute differences | More robust to outliers, but its gradient is fixed, which makes it inefficient at finding the optimal solution. |
| Huber | Lδ = ½(y − ŷ)2 if |y − ŷ| ≤ δLδ = δ|y − ŷ| − ½δ2 otherwise |
— | Combines the best properties of the two: quadratic for smaller errors, linear otherwise. Differentiable everywhere and robust to outliers. The hyper-parameter δ > 0 defines how small the error must be for the loss to stay quadratic; its introduction brings benefits but also makes the loss function more complex. |
±δ and switches to straight lines outside it.Move a single prediction and watch the three losses disagree about how bad it is.
Classification predictive modeling is the task of approximating a mapping function from input variables to discrete output variables called classes, labels or categories. Three losses matter.
Binary cross-entropy (or log loss) is the default loss function to use for binary classification problems. Usually the output value ŷ of a binary classification model represents the probability that the input element belongs to the positive class (1); consequently the probability of the negative class (0) is 1 − ŷ. Given the output label y ∈ {0, 1} and the predicted output ŷ ∈ (0, 1):
L = −log(1 − ŷ) if y = 0
L = −log(ŷ) if y = 1or, written as a single expression:
L = −y · log(ŷ) − (1 − y) · log(1 − ŷ)Both branches are curves that go to infinity as the prediction approaches the wrong end of the interval: an arbitrarily confident wrong answer is arbitrarily expensive.
The generalization of binary cross-entropy to multi-class problems. The output ŷ is a vector where each element ŷi represents the probability that the input sample belongs to the i-th class. Given the labels y encoded as a one-hot vector and the predicted probabilities ŷ:
L = − ∑i=1c yi · log(ŷi)where c is the number of classes. Since y is one-hot, every term but one is zero: the loss is minus the log of the probability the model assigned to the correct class. This is the loss that pairs with a softmax output layer.
Focal loss is an extension of binary cross-entropy that down-weights easy examples and focuses training on hard ones:
Lγ = −(1 − p)γ · log(p)where γ is a focus hyper-parameter and p is the probability that the input element belongs to the correct class:
p = 1 − ŷ if y = 0
p = ŷ if y = 1Intuitively, γ reduces the loss contribution from easy examples and extends the range in which an example receives a low loss. At γ = 0 the modulating factor is 1 and focal loss degenerates into plain binary cross-entropy.
Pair each task with its default loss and be able to justify the pairing: regression → square error (MSE), penalizing distant predictions heavily; binary classification → binary cross-entropy, the stated default; multi-class classification → multi-class cross-entropy with a softmax output, since softmax produces a probability distribution and the one-hot target selects the log-probability of the correct class. Then be ready to name the exception cases: MAE when outliers dominate, Huber when you want both, focal loss when easy examples swamp the hard ones.
net = w₀ + ∑i=1d ini · wi and out = f(net). The bias w₀ allows the activation function to be shifted by adding a constant to the input; it is used to delay the triggering of the activation function. It is drawn as a virtual input in₀ = 1 with its own weight, which is why it adds rather than scales.
A Multi-Layer Perceptron is the most common feed-forward neural network (one where connections do not form a cycle), consisting of three or more layers: an input layer, one or more hidden layers, and an output layer. Fully-connected means each neuron in one layer is connected with every neuron in the following layer.
Watch the training and the generalization (validation) errors. Very good on training but relatively poor on validation → overfitting; fix it by increasing the size of the training set or reducing the complexity of the network. Poor on the training set itself → underfitting; fix it by increasing the complexity or type of the model, or by increasing the training time to minimize the cost function.
A set of techniques that increase the quantity, quality and variability of the training set by adding modified copies of existing data or generating new synthetic data: flipping, translation, rotation, scaling, zooming, stretching, cropping, noise, and changes in brightness and contrast. It is the answer when overfitting calls for a bigger training set but obtaining genuinely new data is not feasible.
Without one, the output signal would be a simple linear function and the neuron could not learn or model complicated data such as images, audio or speech. The activation function introduces non-linearity, allowing the network to learn and compute more complex functions, and it also restricts the output value into a certain finite range.
Nonlinear (a two-layer ANN with a non-linear activation is a universal function approximator); range (finite range means more stable training, infinite range is more efficient but needs a smaller learning rate); continuously differentiable (needed for gradient-based optimization); monotonic (guarantees a convex error surface for a single-layer model); and approximates identity near the origin (efficient learning from small random initial weights).
With ReLU, when inputs approach zero or are negative, the gradient of the function becomes zero and the network cannot learn: the neuron is dead. LeakyReLU (max(0.01x, x)) and ELU (α(ex − 1) for x ≤ 0) both address it by enabling backpropagation even for negative input values, at the price of not providing consistent predictions for negative inputs (LeakyReLU) or slow convergence due to the exponential (ELU).
As a hidden activation it suffers the vanishing gradient problem (for very high or very low inputs there is no change to the prediction), its outputs are not zero centred, and it is computationally expensive. As an output activation its bounded (0, 1) range makes it the natural choice for models where we need to predict a probability.
Softmax maps each output of a layer so that the values lie in (0, 1) and their total sum is 1, so the output is a genuine probability distribution rather than independent probabilities. It is typically used in the output layer to classify inputs into multiple categories, paired with multi-class cross-entropy.
A loss function is for a single training example; a cost function is the average loss over the entire training dataset, C = (∑i=1n L(yi, ŷi)) / n. Optimization strategies aim at minimizing the cost function.
Square error (L2) is the most used; its cost function is MSE; squaring penalizes far-away predictions heavily. Absolute error (L1) gives MAE; it is more robust to outliers but its gradient is fixed, making it inefficient at finding the optimum. Huber is quadratic for errors within ±δ and linear outside; it is differentiable everywhere and robust to outliers, but the extra hyper-parameter δ makes the loss more complex.
It down-weights easy examples so that training focuses on the hard ones. It multiplies the binary cross-entropy by (1 − p)γ, where p is the probability assigned to the correct class. When p is high (an easy example already classified well) the factor is near zero and the example almost stops contributing; γ also extends the range of p over which an example receives a low loss.