Convolutional neural networks

1. Why an MLP fails on images

The chapter opens with a definition from DeepAI: “A Convolutional Neural Network (CNN) is a deep learning neural network designed for processing structured arrays of data such as images.” Today CNNs are used to solve identity recognition, image classification, object detection, scene labeling, visual search, action recognition, document analysis, anomaly detection and video analysis.

But first, why not just use an MLP? The slides give three drawbacks, and the first one is arithmetic.

The parameter explosion

MLPs use one neuron for each input. Take a modest image: 600×400 pixels with 3 colour channels. Flattened, that is

600 · 400 · 3 = 720 000 input neurons

Connect it to a hidden layer of just five neurons and you already have

720 000 · 5 = 3.6 million weights   (for a single layer)

The amount of weights rapidly becomes unmanageable for large images.

No translation invariance

MLPs react differently to an image and its shifted version, because they are not translation invariant. The same cat, moved ten pixels to the right, hits an entirely different set of weights.

Loss of spatial information

Most important of all: spatial information is lost when the image is flattened into an MLP. Pixels that are close together matter, because they help to define the features of an image; flattening throws that adjacency away and the network has to rediscover it from scratch.

2. The visual cortex

In 1968, D. H. Hubel and T. N. Wiesel demonstrated that mammals visually perceive the world around them using a layered architecture of neurons in the brain. As information passes from our eyes to the brain, higher and higher order representations are formed.

Within the visual cortex, complex functional responses generated by complex cells are constructed by combining more simplistic responses from simple cells:

Spatial invariance is obtained by “summing” the contribution of simple cells responding to the same orientation but with different receptive fields. Hold on to that sentence: it is a description of pooling, arrived at thirty years before pooling layers were invented.

3. The architecture of CNNs

This is the inspiration behind CNNs. Higher and higher representations are formed through the layers:

• the early layers, taking in the raw pixels, find edges;
• then more abstract features are found by combining these edges;
• finally, the last layers find higher order semantic meaning.

CNNs were introduced in 1998 by Y. LeCun and Y. Bengio. Their architecture is based on local connections, layering and spatial invariance. The main differences compared with an MLP:

A CNN is a combination of two basic building parts:

• the convolutional part — convolutional and pooling layers. This part forms the essential component of feature extraction.
• the fully-connected part — a fully-connected neural network architecture. This part performs the task of classification based on the input from the convolutional part.

More formally, a CNN is a sequence of layers, and every layer transforms one volume of activations to another through a differentiable function. Three main types of layers build the architecture:

4. Convolution

Convolution is one of the most important image processing operations. A filter strides across the width and height of the input and the Frobenius inner product between the filter and the input is computed at each position — the operation defined in Chapter 2, now put to work.

The slides show that classical image filters are exactly this operation with hand-chosen numbers: an edge detection kernel and a sharpen kernel are just two different 3×3 matrices slid over the image. The insight of a CNN is that those numbers do not need to be chosen at all.

5. The convolutional layer

In a CNN, the objective of the convolution operation is to extract features from the input volume while maintaining the spatial relationship between input elements. Each convolutional layer contains a set of filters (or convolutional kernels).

The shape contract:

The slides give a concrete instance: WI = HI = 32, C = 3, K = 10, F = 5, giving WO = HO = 32.

Local connections and shared weights ensure that different portions of the input volume are processed in the same way. This is a desired behaviour, since different field-of-view regions contain the same type of information.

Counting connections and weights

This distinction is the heart of the chapter. Each element of the output volume is connected to a number of input elements equal to the size of the filter, so the total number of connections is:

(WO · HO · K) · (F · F · C)

But the total number of weights is much smaller, since the weights of each filter are shared by all the elements contained in the same feature map:

(F · F · C + 1) · K        (the +1 is the bias, if present)

6. Stride, padding and output size

Four hyper-parameters control the size of the output volume:

They combine into one formula, which you should be able to write from memory:

WO = (WI − F + 2 · Padding) / Stride + 1

The worked example

The slides take a 5×5 input holding the values 0 to 24 and the kernel

0  1  0
0  1  0
0  1  0

which simply sums the middle column of each 3×3 window, and run it three ways:

Reproduce all three below, then change the kernel and watch which structures each one responds to.

7. Multiple channels and the receptive field

Multi-channel convolution

When the input volume contains multiple channels (C), we use convolutional kernels with the same number of channels. To yield a two-dimensional feature map for each kernel:

1. perform a convolution operation between each input channel and the corresponding channel of the kernel;
2. add the C results all together, summing over the channels.

The slides work a two-channel example with 2×2 kernels: the first channel produces

19  25          37  47
37  43   and    67  77

for the second, and adding them element by element gives the single feature map

 56   72
104  120

Receptive field

The receptive field of a particular feature map unit is the size of the region in the input that influences its activation. In other words, it indicates how much of the input image contributes to the computation of a specific feature at any layer of the CNN.

This is the quantity that grows as you stack layers: a unit two convolutions deep sees a wider patch of the original image than a unit one convolution deep, which is what allows later layers to represent larger structures.

Learned filters

A CNN autonomously learns the kernel weights during the training process. Empirically:

• the first convolutional layer is usually responsible for identifying low-level features such as edges, colour, gradient orientation;
• the subsequent layers allow increasingly higher-level features to be distinguished, giving the network a more detailed understanding of the image.

Which is precisely the simple-cell to complex-cell progression Hubel and Wiesel described in the visual cortex.

8. The pooling layer

Two observations motivate pooling. First, our ultimate task usually asks some global question about the image, so the units of the final layer should be sensitive to the entire input. Second, when detecting lower-level features such as edges, we often want our representations to be invariant to translation.

A pooling layer therefore serves a dual purpose:

• aggregating information to reduce the spatial resolution while maintaining dominant features — which reduces the number of parameters, the computational power required, and the risk of overfitting;
• mitigating the sensitivity of convolutional layers to location, obtaining an approximate translation invariance.

Mechanically, a pooling operator consists of a fixed-shape window that slides over all regions in the input, computing a single output for each location. The crucial difference from convolution:

The slides work both on the same 4×4 input with a 2×2 window and stride 2:

The trade-off between the two operators:

Two structural facts: the pooling layer operates independently on every channel of the input volume, keeping the depth size unchanged; and, as with convolutional layers, pooling layers can change the output shape by padding the input and adjusting the stride.

9. The fully-connected part

Usually the fully-connected part is a simple MLP, consisting of two or three hidden layers and an output layer, that performs the classification among a large number of categories.

The bridge between the two halves is flattening: the output volume of the convolutional part, of size W × H × C, becomes a vector of

m = W · H · C

elements x1, …, xm, which are the inputs of the MLP.

Why bother with dense layers at all after all that convolution? Because adding fully-connected layers is a way of learning non-linear combinations of the high-level features represented by the output of the convolutional part.

Generally, the output layer is implemented using softmax as activation function, to ensure that the sum of the outputs is 1:

softmax(y)i = eyi / ∑j=1n eyj,    y ∈ ℝn

10. 1D CNNs for time sequences

Everything so far assumed images, but the machinery generalises. The key difference between 2D and 1D convolutional layers is the dimensionality of the input/output data and how the filter slides across the data:

1D CNNs are very effective at extracting features from shorter, fixed-length segments of the overall dataset, and in cases where it is not so important where the features are located in the segment. They work well for the analysis of time sequences or any kind of signal data over a fixed-length period, such as accelerometer data or audio signals. Recently 1D CNNs have also been applied to problems in natural language processing, obtaining interesting results.

Check your understanding