Generating is learning a distribution

Classifying versus generating

A model that classifies learns $p(y\mid x)$: given the data, the label; the boundary is enough. A generative model learns the distribution of the data itself, $p_\theta(x)$, and lets you sample $x\sim p_\theta(x)$. It is usually trained by maximum likelihood:

The cloud of sevens

Each digit is measured as $m=(m_1,m_2)$: the ink in its top half and the ink in its bottom half (each pixel is worth 0 to 1; each half has 32 pixels). The lesson’s ovals are a two-dimensional Gaussian fitted by maximum likelihood to the 179 sevens:

The small oval is $(m-\hat\mu)^\top\hat\Sigma^{-1}(m-\hat\mu)=1$ and the large one is $=4$. In a two-dimensional Gaussian, $1-e^{-1/2}\approx 39\%$ and $1-e^{-2}\approx 86\%$ of samples fall inside them. To sample, take $\varepsilon\sim\mathcal N(0,I)$ and $m=\hat\mu+L\varepsilon$, with $LL^\top=\hat\Sigma$ (Cholesky). The boundary between sixes and sevens in the scene is the linear discriminant analysis one: the line where the two Gaussians, with their averaged covariance, give the same density.

Why each pixel on its own is not enough

Drawing each pixel from its own distribution amounts to assuming $p(x)=\prod_{j=1}^{64}p(x_j)$: independent pixels. For digits that is false (a stroke lights up its neighbors), which is why you get smudges. You need a model of how pixels vary together.

Autoencoder

An encoder $z=E_\phi(x)$ and a decoder $\hat x=D_\theta(z)$, trained together to minimize the reconstruction error:

The lesson’s models are tiny: $64\to32\to2\to32\to64$ with $\tanh$ in the hidden layer and a sigmoid at the output (4,354 parameters), trained with gradient descent (Adam) on 1,500 images; another 297 were held back so the network could rebuild images it never saw. Mean error per held-out image by code size: 3.14 with 1 number, 2.34 with 2, 1.46 with 4, 0.95 with 8 and 0.54 with 16 (always guessing the mean image would give 4.71). With a linear decoder and squared error, an autoencoder learns the same subspace as PCA (Goodfellow et al., 2016, §14.1); nonlinear layers generalize it, and Hinton and Salakhutdinov (2006) showed that deep autoencoders learn codes that work much better than PCA.

Variational autoencoder (VAE)

We postulate a code with a known prior, $p(z)=\mathcal N(0,I)$ (the “round zone”), and a decoder $p_\theta(x\mid z)=\mathcal N(D_\theta(z),\sigma_{\text{obs}}^2 I)$. The encoder returns a cloud, $q_\phi(z\mid x)=\mathcal N\big(\mu_\phi(x),\operatorname{diag}\sigma^2_\phi(x)\big)$: its center and its size. During training a point is drawn with the reparameterization trick, which lets the gradient through:

What is maximized is a lower bound on $\log p_\theta(x)$ (the ELBO):

The first term is the reconstruction error; the second is the penalty that pulls each cloud toward the standard zone. For diagonal Gaussians it has a closed form:

The loss used in the lesson is $\lVert x-\hat x\rVert^2/(2\sigma_{\text{obs}}^2)+D_{\mathrm{KL}}$ with $\sigma_{\text{obs}}^2=0.1$ (4,420 parameters). The two terms compete: reconstruction separates and shrinks the clouds; the penalty pulls them together and widens them. To generate: $z\sim\mathcal N(0,I)$ and $x=D_\theta(z)$. With a fixed-variance Gaussian decoder, the best reconstruction of a code is the average of the images that fit it: hence the blur. Kingma and Welling (2019, §2.8.2) attribute it to the fit never being perfect and note that it eases with more flexible inference and generative models.

Sampling with each model

The autoencoder defines no zone to sample from; the naive fallback is to draw uniform points in the box of its codes. A simple classifier (logistic regression trained on the same 1,500 images) labeled 3,000 samples from each model: the autoencoder gave 22% fours, 21% sixes and 0.7% eights; the VAE, between 6% and 14% of each digit (the real data has about 10% of each). With eight different seeds the result holds: the VAE always comes out more even (see train_toy.py).

In code

import numpy as np

# five sevens measured as (top ink, bottom ink): example
m = np.array([[12.0, 7.8], [10.4, 8.9], [11.3, 7.4], [9.6, 8.5], [10.9, 8.1]])
mu = m.mean(axis=0)                        # center of the cloud
cov = np.cov(m.T, bias=True)               # shape of the oval (maximum likelihood: divide by N)
L = np.linalg.cholesky(cov)                # L @ L.T == cov
rng = np.random.default_rng(0)
new = mu + rng.standard_normal((3, 2)) @ L.T   # sampling: three new points from the cloud
print(mu.round(2).tolist(), new.round(1).tolist())
# [10.84, 8.14] [[10.9, 8.0], [11.4, 7.9], [10.4, 8.5]]

# A VAE cloud: center and size (one per number in the code)
c, s = np.array([0.8, -0.3]), np.array([0.5, 0.8])
kl = 0.5 * np.sum(c**2 + s**2 - np.log(s**2) - 1)   # penalty against the standard zone N(0, I)
z = c + s * np.array([0.2, -1.0])                   # a point drawn from the cloud (reparameterization trick)
print(round(float(kl), 4), z.round(2).tolist())
# 0.7263 [0.9, -1.1]

What comes next

VAEs were proposed independently by Kingma and Welling (2013) and by Rezende, Mohamed and Wierstra (2014). Generative adversarial networks (lesson 16) learn to generate without writing down a density: by competing. Diffusion (lesson 17) learns to remove noise, and its latent version works inside the compressed space of an autoencoder.