Generative models

1. Discriminative vs generative modeling

The chapter opens with a definition from Pattern Recognition and Machine Learning: “Approaches that explicitly or implicitly model the distribution of inputs as well as outputs are known as generative models, because by sampling from them it is possible to generate synthetic data points in the input space.”

Until now, this course has mainly talked about making predictions by mapping data examples to labels: this kind of learning is called discriminative learning. Classifiers and regressors are both examples of discriminative learning. But there are machine learning tasks that cannot be solved with discriminative approaches — for example, learning a model to represent the characteristics of a dataset. This kind of learning is called generative modeling: a generative model describes how a dataset is generated, in terms of a probabilistic model, and by sampling from this model we can generate new data.

The fundamental difference:

• discriminative models learn the boundary between classes;
• generative models explicitly/implicitly model the distribution of individual classes.

More formally, given a set of data instances x and a set of labels y:

• discriminative models learn the conditional probability p(y|x) directly from data;
• generative models learn the joint probability p(x, y); the conditional probability can then be predicted using the Bayes Theorem (p(x, y) = p(y|x) · p(x)).

The slides contrast the two on a toy dataset (x, y) : (1,0), (1,0), (2,0), (2,1). The discriminative model estimates p(y|x) directly: for x = 1, p(y=0|x) = 1, p(y=1|x) = 0; for x = 2, both outcomes get ½. The generative model estimates the joint p(x, y): p(x=1, y=0) = ½, p(x=1, y=1) = 0, p(x=2, y=0) = ¼, p(x=2, y=1) = ¼ — a complete description of how the data were produced.

Two examples sharpen the distinction:

• Classify a speech to a language: a discriminative approach determines the linguistic differences without learning any language; a generative approach learns each language and determines which language the speech belongs to.
• Classify an animal as a cat or a dog based on weight and height: a discriminative approach finds a decision boundary that separates cats and dogs and checks on which side the new observation falls; a generative approach builds models of what cats and dogs look like and compares the new observation against the two models.

In terms of practice: discriminative algorithms generally give better performance in classification tasks; generative algorithms can use p(x, y) to generate new data similar to existing data. Examples: discriminative — k-NN, SVM, random forest; generative — the naïve Bayes classifier.

2. Classes of generative models

Generative models can be divided into two categories:

• explicit density models define an explicit density function pmodel(x) similar to pdata(x);
• implicit density models define a stochastic process that, after training, aims to draw samples from the underlying data distribution pdata(x) without explicitly defining it.

The main difficulty in designing an explicit model is capturing all the complexity of the data to be generated while still maintaining computational tractability. Consider modeling the distribution of horse images to generate new full-HD imaginary horses: full HD (1920 × 1080) RGB images have more than 6M dimensions — it is impossible to deal with functions in such a high-dimensional space. Implicit models do not care about the data distribution: their objective is to produce outputs as similar as possible to the real ones.

3. Applications

The application list is broad and overlaps with Chapter 9’s: image denoising, image inpainting (filling in missing regions), image super-resolution, image generation, image-to-image translation, text-to-image synthesis, exploration in reinforcement learning, neural network pretraining, language generation, text-to-speech, imitation learning, and classification.

Two are illustrated in the slides: image super-resolution compares bicubic interpolation with a GAN-based super-resolution model; image-to-image translation turns one depiction of a scene into another. Both will reappear concretely in the Pix2Pix and CycleGAN sections.

4. Deep generative models

Deep generative models are formed through the combination of generative models and deep neural networks. The basic idea is to force the model to discover and efficiently internalize the essence of the data to generate it. The most popular deep generative models are:

• variational autoencoders (explicit density models) — Chapter 9;
• generative adversarial networks (implicit density models) — this chapter;
• diffusion models (implicit density models) — this chapter.

5. Generative adversarial networks

The section opens with a quote from Yann LeCun: “Generative adversarial networks are the most interesting idea in the last 10 years in machine learning.”

A Generative Adversarial Network (GAN) is a model architecture (designed by I. J. Goodfellow and his colleagues in 2014) for training a generative model. GANs rely on the idea that a generator is good if we cannot tell fake data apart from real data. (The slides invite you to guess which of eight photos is fake — the point is that a well-trained generator produces images that pass the test.)

The GAN model architecture consists of two sub-models:

• the generator (G) is trained to produce plausible data (fake);
• the discriminator (D) is trained to distinguish the generator’s fake data from real examples.

The two models are trained together, in an adversarial game, until the discriminator is no longer able to distinguish a real from a fake example. The trajectory: when training begins, the generator produces obviously fake data and the discriminator quickly learns to tell that it is fake; as training progresses, the generator gets closer to producing output that can fool the discriminator; finally, the discriminator starts to classify fake data as real, and its accuracy decreases.

Are GANs supervised or unsupervised?

When we talk about supervised learning, we usually mean learning to predict a label associated with the data, with the goal of generalizing to new data. In GANs, these components are not present: the data come in with no labels, and there is no prediction on new data to generalize. The goal is to model what the data look like (density estimation) and to be able to generate new examples of what has been learned. The slides’ verdict: GANs are unsupervised learning algorithms that use a supervised loss as part of the training — the discriminator’s real/fake prediction is a supervised binary classification task, but the objective it serves is generative.

6. The discriminator

The discriminator tries to distinguish real data from the data created by the generator. Any network architecture appropriate to the type of data to classify can be used. The training data come from two sources:

• real data instances Xi, used as positive examples during training;
• fake data instances X̂j = G(zj), used as negative examples during training.

During discriminator training the generator does not train: its weights remain constant while it produces examples for the discriminator to train on. The discriminator is connected to two loss functions — the discriminator loss (LD) and the generator loss (LG) — but during discriminator training it ignores the generator loss and just uses the discriminator loss. The procedure:

1. the discriminator classifies both real data Xi and fake data G(zj);
2. the discriminator loss LD penalizes the discriminator for misclassifying a real instance as fake or a fake instance as real;
3. the discriminator updates its weights through backpropagation from the discriminator loss.

Since the discriminator performs a binary classification (real vs fake), the binary cross-entropy is used as loss function:

LD = −y · log ŷ − (1−y) · log(1−ŷ)

where ŷ and y are the predicted and the desired outputs (0 = fake and 1 = real). For each iteration, the discriminator is trained on a mini-batch containing m examples from the real dataset X1, …, Xm and m fake examples from the generator G(z1), …, G(zm); the discriminator cost function is:

CD = [ ∑i=1..m −log D(Xi)  +  ∑j=1..m −log(1 − D(G(zj))) ] / (2m)

7. The generator

The generator learns to make the discriminator classify its output as real. Generator training requires tighter integration between the generator and the discriminator: the portion of the GAN that trains the generator includes the random input z, the generator network (which transforms the random input into a fake data instance G(z)), the discriminator network (which classifies the generated data), the discriminator output D(G(z)), and the generator loss LG, which penalizes the generator for failing to fool the discriminator.

The random input

Normally we input data we want to do something with, like an instance to classify or to make a prediction about. But what do we use as input for a network that outputs entirely new data instances? Commonly, a GAN takes random noise as its input: the generator transforms the noise into a meaningful output. Introducing noise lets a GAN produce a wide variety of data, sampling from different places in the target distribution. Experiments suggest the distribution of the noise is not very important, so we can choose something easy to sample from, like a uniform distribution.

Generator training

In the GAN architecture, the generator is not directly connected to the loss: it feeds into the discriminator, and the discriminator produces the output we are trying to affect. The generator loss penalizes the generator for producing a sample that the discriminator classifies as fake; backpropagation starts at the output and flows back through the discriminator into the generator. Moreover, we do not want the discriminator to change during generator training. The procedure:

1. sample random noise (z);
2. produce generator output G(z) from the sampled random noise;
3. get the discriminator real-or-fake classification for the generator output D(G(z));
4. calculate the generator loss LG from the discriminator classification;
5. backpropagate through both the discriminator and generator to obtain gradients;
6. use the gradients to change only the generator weights.

The generator is trained to minimize the probability of the discriminator classifying the data received from the generator as fake. Because only fake data are used during training, a simplified version of the binary cross-entropy can be used as loss function:

LG = log(1 − D(G(z)))

Note that the minus sign is omitted to reduce (and not increase) the probability of the discriminator being correct. Given m fake examples {G(z1), …, G(zm)}, the generator cost function is:

CG = ∑i=1..m log(1 − D(G(zi))) / m

8. Training dynamics and mode collapse

GAN training proceeds in alternating periods:

1. the discriminator D trains for one or more epochs (keeping the generator constant);
2. the generator G trains for one or more epochs (keeping the discriminator constant);
3. repeat steps 1 and 2 to continue training both.

As the generator improves with training, the discriminator performance gets worse because it cannot easily tell the difference between real and fake. If the generator succeeds perfectly, then the discriminator has a 50% accuracy — it is guessing.

One of the main challenges in training GANs is mode collapse. It arises when the generator learns to deceive the discriminator by producing examples from only one specific class, thereby disregarding the full diversity of the training dataset: the generator finds one plausible output that fools the discriminator and keeps producing it.

9. Conditional GANs

Is there any way to provide extra information to the model about what type of output we want to generate? In 2014, M. Mirza and S. Osindero proposed an extension to the GAN architecture (called conditional GAN — cGAN) allowing the data generation process to be conditioned by providing additional information to both generator and discriminator. The additional input (c) can be any kind of auxiliary information, such as class labels or data from other modalities. cGANs are used in a variety of tasks such as text-to-image generation and image-to-image translation.

The architecture is the GAN architecture with one extra input: both the generator and the discriminator receive the additional information c alongside their usual inputs (z for the generator; real/fake data for the discriminator). The slides show that conditioning visibly improves the faithfulness of the generated output compared with the unconditional GAN.

10. Deep convolutional GANs

Deep Convolutional GAN (DCGAN) is a GAN architecture designed in 2015 by A. Radford, L. Metz and S. Chintala for unsupervised learning. Before its introduction, several attempts to improve GANs to model images using CNNs had been unsuccessful primarily due to training instability. The authors defined guidelines for stable DCGAN training:

1. Replace all pooling layers with strided (discriminator) and transposed (generator) convolutions: both networks learn their own spatial downsampling (or upsampling) using convolution.
2. Use batch normalization in all layers except the generator output layer and the discriminator input layer, to stabilize learning.
3. Remove fully-connected hidden layers for deeper architectures (the generator’s first layer may be fully-connected but its result is reshaped into a 3-dimensional tensor and used as the start of the convolution stack; the discriminator’s last convolutional layer is flattened and fed into a single sigmoid output).
4. Activation functions: use ReLU in the generator for all layers except the output, which uses Tanh; use LeakyReLU in the discriminator for all layers.

Transposed convolution

While convolutional and pooling layers often reduce spatial dimensions, in some applications (e.g., semantic segmentation, or a generator that must grow an image) it is necessary to increase input width and height. Transposed convolution is used to upsample the input volume to a desired output shape using learnable parameters.

For a standard convolution with input size Win, kernel F, padding and stride, the output size is Wout = (Win − F + 2·Padding)/Stride + 1. For the transposed convolution the input defines the weight of the kernel, the roles of input and output are reversed, and the kernel contributions are added together where they overlap:

Wout = (Win − 1) · Stride + F − 2 · Padding

DCGAN results, from the slides: generated bedrooms after training on the LSUN bedrooms dataset, and two remarkable properties of the latent space. First, input vector interpolation: interpolating between two input vectors z1 and z2 produces a gradual visual change — the generator maps nearby noise points to nearby images. Second, vector arithmetic: simple arithmetic operations reveal a rich linear structure in representation space, e.g. zman with glasses − zman without glasses + zwoman without glasses yields a vector whose nearest neighbor is the vector for woman with glasses. Experiments on single samples were unstable; averaging the z vector over multiple exemplars produced consistent, stable generations that semantically obeyed the arithmetic.

11. Pix2Pix: paired image-to-image translation

Pix2Pix is a cGAN model proposed in 2016 by P. Isola, J. Zhu, T. Zhou and A. A. Efros for general-purpose image-to-image translation: taking images from one domain and transforming them so they have the style (or characteristics) of images from another domain — colorization, super-resolution, synthesis from segmentation maps.

The architectures employed for the generator and discriminator closely follow DCGAN:

• a U-Net as generator with batch normalization, LeakyReLU and skip connections;
• a PatchGAN as discriminator that only penalizes structure at the scale of patches.

The discriminator: PatchGAN

The discriminator takes an image from the source domain (C) and an image from the target domain (X or X̂ = G(C)) — the input is the concatenation of the two images — and predicts the likelihood that the target-domain image is a real or generated version of the source image. The difference from a regular GAN discriminator: a regular discriminator maps the input to a single scalar output ŷ; a PatchGAN maps the input image to an N × N output volume where each element ŷij signifies whether the patch ij of the input image is real or fake. As in cGANs, binary cross-entropy is used as loss function (alternatively, the squared error).

The generator: U-Net

The generator uses a modified version of an encoder-decoder architecture originally designed for biomedical image segmentation, called U-Net. U-Net has skip connections between encoder and decoder layers: between each layer i and layer n − i (where n is the total number of layers), each skip connection simply concatenates all channels at layer i with those at layer n − i. The encoder has seven convolutional blocks (convolution + LeakyReLU, batch normalization except in the first block); the decoder has seven upsampling convolutional blocks (upsampling + convolution + ReLU, batch normalization everywhere). The upsampling layer is a simple unpooling layer with no trainable parameters that increases the input spatial dimensions using nearest neighbor.

Randomness and loss

The generator does not receive the noise z as input: after initial experiments, the authors noted that the generator simply learns to ignore the noise. Without input noise the generator would be deterministic, so randomness is provided in the form of dropout, applied on several decoder layers at both training and generation time (despite this, the authors observed only minor stochasticity in the output). The generator is trained with a combined loss:

LG = log 𝜇(1 − Ŷ)  +  λ · 𝜇( |X − G(C)| )

where Ŷ = D(C, G(C)) is the output predicted by the discriminator (0 = fake and 1 = real), X and G(C) are the desired and the generated images in the target domain, λ is a hyper-parameter controlling the relative importance of the two components, and 𝜇 is the average of a tensor. The adversarial loss encourages plausible images in the target domain; the L1 loss (absolute error between the generated image and the expected output) encourages faithful translations of the source image — the slides show that L1 alone blurs, adversarial alone invents details, and the combination gives the best of both.

12. CycleGAN: unpaired translation

Training an image-to-image translation model typically requires a large dataset of paired examples of source and target domain images (for example, edge images and the photos they came from). This requirement is a limitation: paired datasets are challenging and expensive to prepare (e.g., photos of different scenes under different conditions), and in many cases they simply do not exist (famous paintings and their photographs). Is it possible to train an image-to-image translation system without a paired dataset? For example, take two collections of horse and zebra photos with unrelated scenes and locations, and translate specific photos from one group to the other. This is the problem of unpaired image-to-image translation.

In 2017, J. Zhu, T. Park, P. Isola and A. A. Efros proposed the Cycle Consistent Adversarial Network (CycleGAN) to perform image-to-image translation without paired examples. Since in an unpaired dataset there is no predefined transformation that can be learned by the generator, the idea is to create such a transformation by enforcing a two-step cycle: the input image is mapped from the source (A) to the target (B) domain, then the obtained image is transformed back from the target (B) to the source (A) domain.

CycleGANs learn at the same time the translation in both directions, involving two DCGANs used as generators (GA→B and GB→A) and two PatchGANs used as discriminators (DA and DB). Given a source domain image (IA), the generated image (ÎB = GA→B(IA)) cannot be compared with a target reference image (as for paired datasets). A meaningful transformation is enforced by comparing the reconstructed image (İA = GB→A(GA→B(IA))) and the original one (IA): ideally they should be identical, so the generators are penalized when the difference is large, using an extra criterion called cycle consistency loss.

The discriminators are trained with square error loss (instead of binary cross-entropy), which the authors noted is more stable during training and generates higher quality results:

CDA = [ ∑i=1..m 𝜇(1 − DA(IA(i)))²  +  ∑j=1..m 𝜇(DA(ÎA(j)))² ] / (2m)

where IA(i) is a real image from domain A, ÎA(j) = GB→A(IB(j)) is a domain-A fake image generated from a domain-B real image, and DA(I) is the discriminator output (0 = fake, 1 = real).

The generators are trained with a combined loss: an adversarial loss (square error, as for the discriminators) that encourages plausible images in the target domain, plus the cycle consistency loss that measures the difference between the output of the second generator and the original image using L1 loss. The cycle consistency loss acts as a regularization of the generators, guiding the generation process toward image translation:

LGA→B = 𝜇(1 − DB(ÎB))²  +  λ · 𝜇( |IA − İA| )

where IA is a real image from domain A, ÎB = GA→B(IA) is a domain-B fake image, İA = GB→A(GA→B(IA)) is the domain-A image reproduced from the domain-B fake, DB(I) is the output of discriminator DB, and λ controls the relative importance of the two components.

13. Diffusion models

The section opens with a definition from Introduction to Diffusion Models for Machine Learning: “Diffusion models work by destroying training data through the successive addition of Gaussian noise and then learning to recover the data by reversing this noising process.”

In both GANs and VAEs, the generator and the decoder respectively transform a latent noise vector into the target distribution in a single step. The idea behind diffusion models: estimating and analysing small step sizes is more manageable than attempting to transform random noise directly into the learned distribution in a single step. A Diffusion Model (DM) typically generates higher-quality data compared to VAE, and is easier to train than GANs and does not suffer from mode collapse.

DMs consist of two key processes:

• Forward process (fixed) — this process gradually adds Gaussian noise to the input image X0;
• Reverse process (generative) — a neural network learns how to recover the original data X̂0 by progressively denoising the noisy input (X̂T).

Forward process

Given a sample X0 from the training set, during the forward process Gaussian noise is incrementally added to the image over T steps, producing a sequence of noisy samples X1, …, XT. The step sizes are controlled by a variance schedule {βt ∈ (0, 1) | 1 ≤ t ≤ T, 0 < β1 < β2 < … < βT < 1}. The distribution of the noisy sample at step t given the sample at step t − 1 is q(Xt | Xt−1) = 𝚩(√(1−βt) · Xt−1; βt · I). As the steps progress, the data sample gradually loses its distinguishable features; when T → ∞, XT is equivalent to pure random noise (𝚩(0, I)).

Thanks to the reparameterization trick, the noisy sample at step t can be computed as Xt = √(1−βt) · Xt−1 + √βt · ε (with ε ~ 𝚩(0, I)), and the forward process can be accelerated by computing Xt in one step:

q(Xt | X0) = 𝚩( √ᾱt · X0 ; (1−ᾱt) · I )
⇒  Xt = √ᾱt · X0 + √(1−ᾱt) · ε

where αt = 1 − βt  and  ᾱt = ∏i=1..t αi

Reverse process

The objective of a DM is to learn how to iteratively reverse the forward process. The reverse transition p(Xt−1 | Xt) is unknown, but it can be approximated using a neural network pθ.

The training loop:

repeat
    select a random sample X₀ from the training set
    select a random step t ∈ {1, …, T}
    sample Gaussian noise ε ~ 𝚩(0, I)
    compute Xt = √ᾱt · X₀ + √(1−ᾱt) · ε
    gradient descent step on pθ minimizing  L = ∥ε − pθ(Xt, t)∥²
until converged

The network is trained to predict the noise ε that was added (given the noisy image and the step t), not the clean image directly.

Data generation: sample Gaussian noise XT ~ 𝚩(0, I), then iteratively denoise for t = T, …, 1 using the update

Xt−1 = 1/√αt · ( Xt − (1−αt)/√(1−ᾱt) · pθ(Xt, t) ) + σt · ε

with fresh Gaussian noise ε at each step (ε = 0 at the last step), and return X̂0: the denoised image. The slides illustrate the loop with T = 100 steps.

Architecture and high resolution

DMs often use a U-Net-like architecture to represent pθ(Xt, t), with the time step t fed in through a time-step embedding. Iteratively processing high-resolution images over multiple time steps is extremely computationally expensive and requires substantial memory (the slides note the cost of running pθ over 1024 × 1024 images at every step).

Latent Diffusion Models (LDMs) solve this by employing a variational autoencoder to apply the diffusion process in low-dimensional space rather than directly in high-dimensional pixel space: the encoder E produces latent representations E(X) from input data X; latent representations are used as the input to a standard DM; the low-dimensional output of the DM is then fed into the decoder D for upsampling to the original image size. The slides close with the model family that emerged from these ideas: BigGAN, VQ-VAE and the Stable Diffusion model.

14. Chapter summary

• Discriminative and generative models solve different problems: discriminative models find the boundary that separates data into classes; generative models model the underlying distribution of the data and can sample from it.
• Generative models divide into explicit density models (an explicit pmodel(x)) and implicit density models (a stochastic process that draws samples without defining the distribution).
• GANs train together a generator and a discriminator in an adversarial game until the discriminator can no longer distinguish real from fake; they do unsupervised learning by setting up a supervised learning problem.
• cGANs condition both generator and discriminator with additional information (e.g., a class label) to generate specific output types.
• DCGAN is the convolutional GAN whose guidelines (strided/transposed convolutions, batch normalization, no FC hidden layers, ReLU/Tanh/LeakyReLU) made image GANs trainable; its latent space supports interpolation and vector arithmetic.
• Pix2Pix is a cGAN for paired image-to-image translation with a U-Net generator and a PatchGAN discriminator, trained with adversarial + L1 loss.
• CycleGAN performs unpaired image-to-image translation using two generators and two discriminators plus a cycle consistency loss.
• Diffusion models incrementally transform random noise into the learned distribution through a fixed forward noising process and a learned reverse denoising process; they generate higher-quality data than VAEs and are easier to train than GANs, avoiding mode collapse.
• Latent diffusion models use a VAE to run the diffusion process in a low-dimensional latent space, then upsample with the VAE decoder.

Check your understanding