In 1998, LeNet-5 was introduced to recognize handwritten digits in images. Its layer list is worth memorising because every later architecture is a variation on it:
| Stage | Layers |
|---|---|
| Convolutional | Three convolutional layers: C1, C3, C5 |
| Sub-sampling | Two average pooling layers: S2, S4 |
| Classification | Two fully-connected layers: F6 and Output |
The full pipeline as the slides draw it: Conv (6 kernels 5×5) → Avg pool 2×2 stride 2 → Conv (16 kernels 5×5×6) → Avg pool 2×2 stride 2 → Conv (120 kernels 5×5×16) → Full connection → Full connection.
Its significance and its numbers:
In 2012, AlexNet was introduced, a CNN with an architecture similar to LeNet-5: five convolutional layers, three max pooling layers, three fully-connected layers.
It won the ImageNet Large Scale Visual Recognition Challenge (LSVRC) 2012 by a very large margin, with a top-5 error rate of 15.3% compared with 26.2% for the runner-up. Its historical importance is stated in one sentence by the slides:
AlexNet showed, for the first time, that learned features can overcome hand-crafted features. This is the empirical proof of the claim made in Chapter 1: the hidden layers are a better feature engineer than a domain expert.
The significant differences with respect to LeNet-5:
p = 0.5) is used in the first two fully-connected layers to reduce overfitting;It contains about 60M trainable parameters — a thousand times LeNet-5.
It has been demonstrated that deep neural networks trained on a relatively small dataset can overfit the training data: the model tends to learn the statistical noise in the training data but is not able to generalize, obtaining poor performance on new data.
Dropout is a regularization method useful to avoid overfitting by reducing interdependent learning among the neurons. The basic idea is to pass each training example through a different network architecture, increasing the robustness of the learned parameters.
| Phase | What happens |
|---|---|
| Training | For every training example, each neuron has a probability p of being ignored (dropped out) together with all its incoming and outgoing connections. |
| Testing | All the neurons and the corresponding activations are used, but reduced by a factor p to take into account the missing neurons during training. |
The cost, stated plainly by the slides: dropout roughly doubles the number of epochs required to converge. However, training time for each epoch is less, since fewer neurons are active.
In 2014 the Visual Geometry Group (VGG) proposed VGGNet. The VGG-16 version consists of 13 convolutional layers, five max pooling layers and three fully-connected layers.
Its contribution is compositional rather than numerical. The convolutional part connects several VGG blocks in succession, where a VGG block is a sequence of 3 × 3 convolutional layers followed by a max pooling layer, defined by just two numbers:
Two justifications the slides give for the fixed small kernel:
3 × 3 fixed size kernels reduces the number of parameters;Dropout (p = 0.5) is again used in the first two fully-connected layers. Four versions share the number of pooling and fully-connected layers but differ in the number of convolutional layers:
| Version | Convolutional layers |
|---|---|
| VGG-11 | 8 |
| VGG-13 | 10 |
| VGG-16 | 13 |
| VGG-19 | 16 |
VGG-16 reached a top-5 error rate of 7.3% on ImageNet LSVRC-2014 and contains about 138M trainable parameters — the largest count in this chapter.
# a VGG block, as the slides define it: n conv layers of 3x3, then one max pool
def vgg_block(x, num_convs, channels):
for _ in range(num_convs):
x = Conv2D(channels, kernel_size=3, padding='same', activation='relu')(x)
return MaxPooling2D(pool_size=2, strides=2)(x)
# VGG-16: five blocks with a growing number of output channels
x = vgg_block(inputs, 2, 64)
x = vgg_block(x, 2, 128)
x = vgg_block(x, 3, 256)
x = vgg_block(x, 3, 512)
x = vgg_block(x, 3, 512)In 2014 GoogLeNet won the ImageNet LSVRC-2014 challenge by exploiting three ideas: the 1×1 convolutional layer, global average pooling at the end of the network instead of fully-connected layers, and repeated inception modules.
Its composition: three convolutional layers, four max pooling layers, nine inception modules, a global average pooling layer and a fully-connected layer. It reached a top-5 error rate of 6.7% with only about 7M trainable parameters — twenty times fewer than VGG-16, with a better score.
A 1×1 convolution is used as a dimension reduction module to reduce the computation. By reducing the computation bottleneck, depth and width can be increased. The slides prove the point arithmetically.
Suppose we need to perform a 5×5 convolution on a 14 × 14 × 480 volume with 48 kernels of size 5 × 5 × 480 and padding 2, producing 14 × 14 × 48:
14 · 14 · 48 · 5 · 5 · 480 = 112.9M connectionsInstead, execute a 1×1 convolution first, with 16 kernels of size 1 × 1 × 480 to obtain 14 × 14 × 16, and only then the 48 kernels of size 5 × 5 × 16 with padding 2:
14 · 14 · 16 · 1 · 1 · 480 + 14 · 14 · 48 · 5 · 5 · 16 = 5.3M connections
1×1 convolution 5×5 convolutionThe idea of the 1×1 convolutional layer is to preserve the spatial dimensions of the input volume while reducing its depth. Same output shape, a twentieth of the computation.
The basic block in GoogLeNet consists of four parallel paths:
Finally, the outputs of each path are concatenated along the channel dimension.
The idea behind the module: visual information is processed at various scales and then aggregated, so that the next stage can extract features from different scales simultaneously. For the first inception module the volumes are: input 28×28×192, path outputs of 28×28×64, 28×28×128, 28×28×32 and 28×28×32, concatenated into 28×28×256.
In previous CNNs, fully-connected layers are used at the end of the network and all inputs are connected to each output. In GoogLeNet, global average pooling is used instead, averaging each feature map from 7×7 to 1×1. The saving is total:
| Fully-connected | Global average pooling | |
|---|---|---|
| Weights (connections) | 1024 · 7 · 7 · 1024 = 51.4M | 0 |
That single substitution explains most of the gap between VGG-16 (138M parameters) and GoogLeNet (7M).
Training deep neural networks is difficult, and getting them to converge in a reasonable amount of time can be complicated. Batch Normalization (BN) is an effective technique that consistently accelerates and stabilizes the convergence of deep neural networks:
BN was proposed to overcome the internal covariate shift problem. The slides build the concept with an example: a flowers dataset and a binary classifier for roses. Consider two subsets of the training data with very different distributions, lying in different regions of the feature space. This difference in distribution is called covariate shift, and it makes the training of the classifier very slow.
For the input layer this problem is easily solved by randomizing the data before creating mini-batches. But since an ANN updates its weights during training, covariate shift can occur not only at the input layer but also in hidden layers, forcing them to continuously adapt to changing input distributions. This is the internal covariate shift, and it slows down training and requires a very small learning rate and a good parameter initialization.
The basic idea is to limit internal covariate shift by normalizing the activations of each layer, before the activation function, for each mini-batch, transforming the inputs to have mean 0 and unit variance. This allows each layer to learn on a more stable distribution of inputs, accelerating the training process.
BN adds two trainable parameters, γ and β, to each layer, to let the network decide whether or not to apply the normalization. The identity is recovered exactly when γ = √(σ²B + ε) and β = μB.
The slides give a numeric example over the channels of a mini-batch, with ε = 10−3, γ = 1, β = 0, and per-channel statistics μ1 = 3.13, σ²1 = 3.11; μ2 = 1.50, σ²2 = 0.75; μ3 = 1.88, σ²3 = 0.61. Work one of them below.
μ and σ². To overcome this, μ and σ² are accumulated on the entire training set during the training phase.The slides end the section with an honest correction: although BN is a very useful technique, it has not yet been demonstrated that it effectively reduces the internal covariate shift. Instead, it was discovered that it smooths the optimization landscape, allowing larger learning rates to quickly converge to more accurate solutions. The technique works; the original explanation for why it works is in doubt.
In 2015, ResNet (Residual neural Network), proposed by Microsoft Research, won the ImageNet LSVRC-2015 challenge. It consists of a convolutional layer, a max pooling layer, four residual blocks (B1, B2, B3, B4), a global average pooling layer and a fully-connected layer.
The core idea is the identity shortcut connection (also called residual or skip connection) that skips one or more layers. Its three benefits, as the slides list them:
Two structural choices complete the design: batch normalization is applied after each convolutional layer, and spatial reduction is achieved using a stride equal to 2 in some convolutional layers instead of employing pooling layers.
A residual block contains a variable number of sub-blocks with a series of convolutional layers with a constant number of kernels (64, 128, 256 and 512). Within a residual block, the spatial dimensions of the feature maps remain constant.
Each sub-block contains two convolutional layers and a skip connection that brings the input up to the output, where it is added with the output of the second convolutional layer before the activation function.
Some sub-blocks perform spatial reduction using convolutional layers with stride 2. In such cases the skip connections (drawn dashed in the slides) must reduce their output spatial size before summing the feature maps together, which is done with a 1×1 convolutional layer with stride 2, followed by batch normalization.
There are five versions sharing the overall structure but with residual blocks presenting a different architecture and number of sub-blocks: ResNet-18, ResNet-34, ResNet-50, ResNet-101 and ResNet-152.
ResNet won ImageNet LSVRC-2015 overtaking for the first time a human expert:
| Competitor | Top-5 error rate |
|---|---|
| Human expert | 5.1% |
| ResNet-50 | 5.3% |
| ResNet-152 | 4.5% |
ResNet-50 contains about 25M trainable parameters.
ImageNet is a large visual database designed for use in visual object recognition software, containing more than 14 million images collected from the web and labeled by humans.
The ImageNet Large Scale Visual Recognition Challenge (ILSVRC) is an annual international competition to evaluate image classification and object detection algorithms using that database:
| Quantity | Size |
|---|---|
| Classes | 1000 |
| Training images | 1.2M |
| Validation images | 50K |
| Test images | 100K |
Architectures are compared using Top-1 and Top-5 accuracy, evaluating only the central crop of the ImageNet validation set.
Everything in this chapter, on one line each:
Understanding why a CNN makes a certain prediction is crucial in many applications. Interpretability methods help us visualize which parts of the input a CNN focuses on to produce a given output.
The canonical instrument is the saliency map, which highlights which pixels of the input image most influence a CNN prediction. It is typically computed as the gradient of the output with respect to the input.
Notice what a saliency map reuses: exactly the quantity ∂out/∂ini that Chapter 4 introduced as the thing a layer hands backwards so training can continue. Run the backward pass all the way to the pixels instead of stopping at the first layer, and the same machinery that trains the network also explains it.
More generally, interpretability methods allow us to peek into the decision process, revealing which parts of the input most strongly influence the model predictions.
It is the first ANN to use convolutional and pooling layers to extract spatial features (1998, for handwritten digit recognition). It has three convolutional layers (C1, C3, C5), two average pooling layers (S2, S4) and two fully-connected layers (F6 and Output), uses tanh in C1, C3, C5 and F6, contains about 60K trainable parameters and achieved 99% accuracy on MNIST.
It won ILSVRC-2012 with a top-5 error of 15.3% against 26.2% for the runner-up, showing for the first time that learned features can overcome hand-crafted features. The differences: much deeper; ReLU instead of tanh; local response normalization after the first two convolutional layers; dropout with p = 0.5 in the first two fully-connected layers; and data augmentation. About 60M trainable parameters.
During training, for every training example each neuron has a probability p of being ignored, together with all its incoming and outgoing connections, so each example passes through a different network architecture. During testing, all the neurons and their activations are used but reduced by a factor p to account for the neurons missing during training. It roughly doubles the number of epochs needed to converge, though each epoch is faster.
A VGG block is a sequence of 3×3 convolutional layers followed by a max pooling layer, defined by the number of convolutional layers (1 to 4) and the number of output channels (64, 128, 256 or 512). Fixed 3×3 kernels reduce the number of parameters, and several layers of deep and small kernels are more effective than fewer layers of wider convolutions, because multiple nonlinear layers allow more complex features to be learned.
A direct 5×5 convolution from 14×14×480 with 48 kernels of 5×5×480 and padding 2 costs 14·14·48·5·5·480 = 112.9M connections. Inserting a 1×1 convolution with 16 kernels of 1×1×480 first, then 48 kernels of 5×5×16, costs 14·14·16·1·1·480 + 14·14·48·5·5·16 = 5.3M. The 1×1 layer preserves the spatial dimensions while reducing the depth.
The first three use convolutional layers with window sizes 1×1, 3×3 and 5×5 to extract information from different spatial sizes; the middle two paths perform a 1×1 convolution first to reduce complexity; the fourth uses a 3×3 max pooling layer followed by a 1×1 convolutional layer to change the number of channels. All paths use appropriate padding so the outputs share the same spatial dimensions, and the four outputs are concatenated along the channel dimension.
Averaging each feature map from 7×7 to 1×1 has zero weights, against 1024 · 7 · 7 · 1024 = 51.4M connections for the fully-connected alternative. This is the main reason GoogLeNet needs only about 7M parameters against VGG-16’s 138M.
Covariate shift is a difference in distribution between subsets of the data, which makes training slow; at the input layer it is solved by randomizing the data before creating mini-batches. Since a network updates its weights during training, the same shift occurs in hidden layers, forcing them to continuously adapt to changing input distributions: this is internal covariate shift. BN normalizes the activations of each layer, before the activation function, for each mini-batch, transforming them to mean 0 and unit variance.
They are two trainable parameters added to each layer, to let the network decide whether or not to apply the normalization: the identity is recovered when γ = √(σ²B + ε) and β = μB. At prediction time the model may classify a single image, making mini-batch statistics impossible to compute, so μ and σ² are accumulated over the entire training set during training and reused.
No. The slides are explicit: although BN is very useful, it has not yet been demonstrated that it effectively reduces the internal covariate shift. What was discovered instead is that it smooths the optimization landscape, allowing larger learning rates to converge quickly to more accurate solutions.
(1) It avoids the vanishing gradient problem by creating an alternate shortcut to directly backpropagate the gradient to earlier layers, which is what makes very deep networks trainable. (2) It allows the model to learn an identity function, ensuring a layer performs at least as well as the previous one. (3) Early in training it simplifies the network by letting information pass directly and bypass intermediate layers, which are gradually restored as the network learns weights.
Each sub-block contains two convolutional layers and a skip connection that brings the input up to the output, where it is added with the output of the second convolutional layer before the activation function. Batch normalization follows each convolutional layer. When a sub-block performs spatial reduction with stride 2, the skip connection must reduce its spatial size too, using a 1×1 convolutional layer with stride 2 followed by batch normalization.
Because ResNet overtook a human expert for the first time. The top-5 error rates: human expert 5.1%, ResNet-50 5.3%, and ResNet-152 4.5%. ResNet-50 contains about 25M trainable parameters.
A saliency map highlights which pixels of the input image most influence a CNN prediction. It is typically computed as the gradient of the output with respect to the input — the same backward-pass quantity used during training, propagated all the way to the pixels.