The chapter opens with a definition from Wikipedia: “An autoencoder is a type of artificial neural network used to learn efficient data representations in an unsupervised manner.”
The aim of an AutoEncoder (AE) is to learn a representation (encoding) for a set of data, typically for dimensionality reduction, by training the network to ignore signal “noise”. AEs transform the input into a new representation — called code or latent-space representation — and then reconstruct the output from this representation.
Formally, an AE is composed of two functions:
E(x): ℝn → ℝk outputting a latent representation s;D(s): ℝk → ℝn computing the reconstructed output o.The whole pipeline reads x → E(x) → s → D(s) → o: the input is compressed into the code and then expanded back into a reconstruction of the input.
| Characteristic | Meaning |
|---|---|
| Data-specific | AEs are only able to compress data similar to what they have been trained on. |
| Lossy | The output of an AE will not be the same as the input: it will be a close but degraded representation. |
| Unsupervised | AEs do not need explicit labels to train on. To be more precise they are self-supervised, because they generate their own labels from the training data (the input itself is the label). |
Principal Component Analysis performs the same dimensionality-reduction task with a linear projection, so the question is fair. The slides list four answers:
s (vermilion bottleneck), the decoder expands it back. Training minimizes the reconstruction error between x and o, forcing the code to keep what matters.The application list doubles as a map of what representation learning buys you: clustering, anomaly detection, dimensionality reduction, data denoising, classification, data reconstruction, data generation, machine translation, information retrieval and recommendation systems.
Two of these deserve an early note. Anomaly detection works because an AE trained on normal data reconstructs normal inputs well and abnormal inputs poorly — the reconstruction error is the anomaly score. And data generation is the one that does not come for free: a plain AE is not a generative model, as section 9 shows in detail.
The simplest form of an AE is a feed-forward neural network having an input layer and an output layer with the same number of neurons and one or more hidden layers connecting them. The purpose is to minimize the difference between the input and the output.
An AE consists of two parts: the encoder (E), which compresses the input x and produces the code s, and the decoder (D), which reconstructs the input o starting from the code s. The AE is trained by minimizing the reconstruction error ℒ(x, o), which measures the difference between the input x and its reconstruction o.
There are four hyper-parameters to set before training:
The training itself is performed through backpropagation, like a regular feed-forward neural network — the machinery of Chapters 4 and 5 applies unchanged.
If the only purpose of an AE is to copy the input to the output, it would be useless. The hope is that during training the latent representation will take on useful properties. The risk is that the AE could learn the so-called identity function, so that the output equals the input — and then it performs no useful representation learning or dimensionality reduction: the code is just a pass-through of everything.
An AE with a code as wide as the input, and enough capacity, can learn o = x trivially. The reconstruction error reaches zero and the network is technically perfect — and completely useless, because the code has not been forced to discard anything.
To avoid this risk, the simplest solution is to use a bottleneck layer which forces a compressed knowledge representation of the original input, constraining the amount of information that can traverse the full network (k < n). Because the code cannot carry the full input, the network is forced to keep only what is needed to reconstruct it — and that is precisely the useful representation.
k < n cannot, so the network must learn to keep only what matters for reconstruction.Several variants exist to the basic model, with the aim of forcing the learned representation to assume useful properties: denoising autoencoders, sparse autoencoders, contractive autoencoders, variational autoencoders, and conditional variational autoencoders.
Denoising AEs prevent the network from learning the identity function by corrupting the input data on purpose — adding noise or masking some of the input values — and making it recover the original noise-free data. The AE cannot simply copy the input to its output: the input it receives is not the target, so it is forced to extract useful features that constitute better higher-level representations of the input.
The input corruption is performed only during the training phase. Once the model has learnt the optimal parameters, in order to extract the representations from the original data, no corruption is added.
Sparse AEs represent an alternative method to avoid the identity function without a reduction in the number of neurons, by using a sparsity constraint. A penalty term is added to the loss function such that only a fraction of the neurons become active:
ℒ(x, o) + Ω(h)where Ω is a penalty function on hidden layer activations (h). This forces the AE:
A sparse AE is forced to selectively activate regions of the network depending on the input data: different inputs result in activations of different neurons. The sparsity of activation can be achieved by formulating the penalty term Ω using either L1 regularization or Kullback-Leibler (KL) divergence.
In L1 regularization, Ω penalizes the sum of the absolute values of hidden layer activations:
Ω(h) = λ · ∑i=1..m |hi|where λ is a scaling hyper-parameter and hi is the hidden layer activation obtained with the i-th training example. The absolute value pushes activations toward zero: most neurons stay silent for each input.
KL divergence is a measure of the difference between two probability distributions. Given the average activation of hidden neuron j, computed by summing the activations of m training observations denoted individually as x(i):
ρ̂j = (1/m) · ∑i=1..m hj(x(i))the objective is to (approximately) enforce the constraint ρ̂j = ρ, where ρ is a sparsity hyper-parameter (e.g., 0.05), a value close to zero that encourages most neurons to be inactive.
Supposing ρ and ρ̂ describe two Bernoulli random variable distributions — “the probability distribution of a random variable which takes the value 1 with probability ρ and the value 0 with probability 1 − ρ” — then Ω can be computed as the KL divergence between the two Bernoulli distributions, summed over all k hidden neurons:
Ω(h) = λ · ∑j=1..k KL(ρ ‖ ρ̂j)
= λ · ∑j=1..k [ ρ log(ρ/ρ̂j) + (1−ρ) log((1−ρ)/(1−ρ̂j)) ]where k is the number of neurons in the hidden layer and λ controls the weight of the sparsity penalty term.
KL divergence has the property that KL(ρ ‖ ρ̂j) = 0 if ρ̂j = ρ, and otherwise it increases monotonically as ρ̂j diverges from ρ. In other words: the penalty is zero exactly when the observed average activation matches the target sparsity, and grows the further the neuron drifts from it.
One would expect that for very similar inputs, the learned encoding would also be very similar. The objective of a contractive autoencoder is to learn a robust representation invariant to unimportant transformations for the training data. In other words, contractive autoencoders make the latent space representation less sensitive to small variations in the training dataset.
This is achieved by adding a term to the loss function to penalize large derivatives of the hidden layer activations with respect to the training examples, penalizing instances where a small change in the input leads to a large change in the encoding space. The penalty term corresponds to the squared Frobenius norm of the Jacobian matrix for the hidden layer activations with respect to the input observations:
ℒ(x, o) + λ · ∑i=1..m ∥∇hi∥2Fwhere λ is a scaling hyper-parameter and hi is the hidden layer activation obtained with the i-th training example.
The benefit is that similar inputs are contracted to a constant output within a neighborhood, based on what the model observed during training. The name “contractive” is literal: the map shrinks small input perturbations into even smaller encoding perturbations.
Given an AE, can new data be generated by decoding points that are randomly sampled from the latent space? The quality and relevance of generated data depend on the regularity of the latent space.
To make the generative process possible, the latent space must satisfy two requirements:
Unfortunately, it is very difficult (if not impossible) to ensure, a priori, that the latent space created by an encoder satisfies these requirements. Because of overfitting, the latent space of an AE can be extremely irregular: points between the clusters learned from the data decode into meaningless outputs, and adjacent points can decode into wildly different contents. This is the exact problem variational autoencoders were designed to solve.
A Variational AutoEncoder (VAE) is an AE whose training is regularized to avoid overfitting and ensure that the latent space satisfies the continuity and completeness requirements, enabling the generative process.
To regularize the latent space, instead of encoding an input as a single point, each input is encoded into the parameters of a k-dimensional multivariate normal distribution — a mean μ and a covariance matrix Σ — over the latent space of size k.
The model is trained as follows:
x) is encoded as a distribution over the latent space (μ(x), Σ(x));s) in the latent space is sampled from the normal distribution 𝚩(μ(x), Σ(x));s) is decoded (os) and the reconstruction error is computed;The only fact that VAEs encode inputs as distributions instead of simple points is not sufficient to ensure continuity and completeness. Without a well-defined regularization term, the model can learn to ignore the distributions and act like a classic AE by returning:
To avoid these effects, both the covariance matrix and the mean need to be regularized, by enforcing the distributions to be close to a standard normal distribution (with mean zero and covariance matrix equal to the identity matrix). In this way:
The loss function is composed of two terms:
The regularization term is expressed as the KL divergence between the returned distribution and a standard Gaussian (𝚩(0, I)):
ℒ(x, os) + λ · KL( 𝚩(μx, Σx) ‖ 𝚩(0, I) )where λ is a scaling hyper-parameter, 0 is the zero vector and I is the identity matrix (both of size k). The KL divergence can be computed in closed form as:
KL( 𝚩(μx, Σx) ‖ 𝚩(0, I) ) = ½ [ μxTμx + tr(Σx) − log|Σx| − k ]Continuity and completeness obtained with regularization tend to create a “gradient” over the information encoded in the latent space. For example, a point of the latent space halfway between two encoded distributions should be decoded into something that is somewhere between the data that gave the first distribution and the data that gave the second one.
To simplify the computational complexity, we can assume that the k dimensions of the latent space are not correlated with each other, and the covariance matrix Σx has nonzero values only on the diagonal. In other words, instead of encoding an input as a k-dimensional multivariate normal distribution, each input is encoded into k univariate normal distributions. This allows describing Σx as a vector σx of length k containing the standard deviation of each univariate distribution. A random point can then be sampled by applying the reparameterization trick:
s = σx ⊙ ε + μx , ε ~ 𝚩(0, 1)where 0 and 1 are the zero and all-ones vectors, and the KL divergence is computed as:
KL( 𝚩(μx, σx) ‖ 𝚩(0, 1) ) = ½ ∑i=1..k ( μi² + σi² − log σi² − 1 )During training, the encoder encodes the input sample into a distribution in the latent space, then a point is randomly sampled from this distribution and the output is reconstructed by the decoder. We need to be very careful about the way we sample: we cannot simply use a random sampling process, because the sampling process must be expressed in a way that allows the error to be backpropagated through the network. A raw random draw has no gradient path back to μ and Σ.
Fortunately, to make gradient descent possible despite the random sampling, the reparameterization trick can be used: if s is a random variable following a normal distribution with mean μx and covariance Σx, then it can be expressed by a random variable ε sampled from a standard normal distribution (𝚩(0, I)), scaled by σx and shifted by μx:
ε ~ 𝚩(0, I), s = σx ⊙ ε + μxThe randomness is isolated in ε, which does not need a gradient; the parameters μx and σx enter through deterministic, differentiable operations (scaling and shifting).
ε; μx and σx are moved out of the sampling step into deterministic operations, so the reconstruction error can be backpropagated to the encoder.The slides walk through an MNIST-style example. The encoder takes in handwritten digit images and produces probability distributions in the latent space; the decoder can produce reasonable handwritten digit images given sampled points from the latent distribution. The picture to keep: points sampled from one region of the latent space decode into one kind of digit, and the regions are organized smoothly enough that the space between them still decodes into meaningful, interpolated content.
A VAE cannot generate specific data (e.g., a particular number on demand) by decoding a point randomly sampled from the latent distribution. It is because the encoder models the latent space directly based on the input, not caring about its type; similarly, the decoder models the output based only on the point sampled from the latent distribution.
A Conditional Variational AutoEncoder (CVAE) is a VAE with an extra input to both the encoder and the decoder to shape the entire generative process on a specific input. At training time, the input type y (i.e., the class, label or category) is provided to both the encoder and decoder. To generate a specific output, the desired type is fed into the decoder along with a random point sampled from the latent distribution.
If the same latent point is fed in to produce two different outputs, the process works correctly, since the system no longer relies on the latent space to encode the type. In a CVAE, the latent space encodes other information: in the handwritten digit example, it could encode information such as stroke width or the angle at which the number is written. The loss function becomes:
ℒ(x, os,y) + λ · KL( 𝚩(μx,y, Σx,y) ‖ 𝚩(0, I) )where x is the input and y its type, μx,y and Σx,y are the mean and covariance of the latent distribution conditioned on the type, and os,y is the decoded output depending on both the sampled point s and the input type y.
AEs are often trained with a single-layer encoder and a single-layer decoder, but using deep encoders and decoders offers many advantages:
These are the same depth arguments from Chapter 3, now applied to the compression objective: a deep encoder can build the representation hierarchically instead of in one giant linear step.
k < n) which forces a compressed knowledge representation of the original input.An AE is a type of artificial neural network used to learn efficient data representations in an unsupervised manner, composed of an encoding function E(x): ℝn → ℝk and a decoding function D(s): ℝk → ℝn. It is data-specific (only compresses data similar to its training data), lossy (the output is a close but degraded representation of the input) and unsupervised — more precisely self-supervised, since it generates its own labels from the training data.
An AE can learn non-linear transformations with non-linear activations and multiple layers; it is more efficient to learn several layers than one huge transformation; it can use pre-trained layers for transfer learning; and it can process images, videos or series data using convolutional layers.
The number of neurons in the middle layer (code size); the number of layers in both encoder and decoder; the number of neurons per layer; and the loss function (e.g., MSE). Training then proceeds through ordinary backpropagation.
If the only purpose of the AE is to copy input to output, it is useless: the network could learn the identity function, so the output equals the input and no useful representation or dimensionality reduction happens. The simplest solution is a bottleneck layer that forces a compressed representation by constraining the amount of information that can traverse the network (k < n).
It corrupts the input data on purpose (adding noise or masking values) and trains the AE to recover the original noise-free data: the AE cannot simply copy its input because the input is not the target, so it must extract useful features. The corruption is performed only during training; at inference on original data no corruption is added.
L1 regularization: Ω(h) = λ∑|hi|, which pushes activations toward zero. KL divergence: given the average activation ρ̂j of hidden neuron j, penalize its deviation from the target sparsity ρ with KL(ρ ‖ ρ̂j), treating both as Bernoulli distributions; the penalty is zero exactly when ρ̂j = ρ and grows monotonically as they diverge.
It adds to the loss a term penalizing large derivatives of the hidden layer activations with respect to the inputs — the squared Frobenius norm of the Jacobian ∥∇hi∥F². This makes the encoding less sensitive to small input variations: similar inputs are contracted to a constant output within a neighborhood, producing robust representations.
Continuity: close latent points should not decode into completely different contents. Completeness: any latent point should decode into meaningful content. A plain AE cannot guarantee either a priori: due to overfitting its latent space is extremely irregular, with gaps between clusters that decode into meaningless outputs.
Instead of encoding an input as a single point, the encoder returns the parameters of a multivariate normal distribution (μx, Σx), and the loss adds a regularization term that pulls every returned distribution close to 𝚩(0, I). The loss is reconstruction error + λ·KL(𝚩(μx,Σx) ‖ 𝚩(0, I)). The KL term prevents punctual distributions (covariance near identity) and means too far apart (mean near zero).
Sampling from the encoder’s distribution is a random operation with no defined gradient, so the reconstruction error could not be backpropagated through it. The trick rewrites the sample as s = σx ⊙ ε + μx with ε ~ 𝚩(0, I): the randomness is isolated in ε, while μx and σx enter through deterministic, differentiable operations, so gradients flow to the encoder.
A VAE encoder models the latent space based on the input without caring about its type, and the decoder works only on the sampled point, so the type cannot be requested on demand. A CVAE feeds the input type y to both encoder and decoder at training time and to the decoder at generation time: the latent space no longer needs to encode the type, and can instead encode other attributes (stroke width, angle).
Depth can reduce the computational cost of representing some functions, decrease the amount of training data needed to learn some functions, and experimentally deep AEs yield better compression compared to shallow or linear AEs.