Diffusion: from noise to image

The forward process

With a noise schedule $\beta_1,\dots,\beta_T$, let $\alpha_t = 1-\beta_t$ and $\bar\alpha_t = \prod_{s\le t}\alpha_s$. Each step is $q(x_t\mid x_{t-1}) = \mathcal N\big(\sqrt{\alpha_t}\,x_{t-1},\ \beta_t I\big)$, and chaining them gives a closed form for any step:

The lesson’s bar shows $\bar\alpha_t$ (the share of the mix’s “energy” that is still image) and $1-\bar\alpha_t$ (the share that is noise). The lesson uses $T = 50$ and the cosine schedule of Nichol and Dhariwal (2021), $\bar\alpha(t) \propto \cos^2\!\big(\tfrac{t/T + s}{1+s}\cdot\tfrac{\pi}{2}\big)$ with $s = 0.008$, clipping each $\beta_t$ to at most 0.35 (with so few steps, the last cosine $\beta$ values approach 1). So $\bar\alpha$ is 0.899 at step 10, 0.494 at 25, 0.094 at 40, and 0.003 at 50. Pixels are first mapped to the range $-1\ldots 1$; on screen they are clipped to $0\ldots 1$.

Why not in one jump

The network is trained with squared error, and what minimizes squared error is the conditional mean $\mathbb E[x_0\mid x_t]$. At $t = T$ the static keeps almost no information, so $\mathbb E[x_0\mid x_T]\approx \mathbb E[x_0]$: the average of all the digits, the lesson’s smudge. By contrast, if each $\beta_t$ is small, the reverse step $q(x_{t-1}\mid x_t)$ is nearly Gaussian, with the same form as the forward step (Sohl-Dickstein et al., 2015, citing Feller, 1949): estimating its mean is enough.

What the network learns and how it is trained

The network $\varepsilon_\theta(x_t, t, c)$ estimates the noise. The simplified loss of Ho et al. (2020) is

Thanks to the closed form, training does not need to walk through the steps: it jumps straight to a random $t$. The lesson’s pixel network has 53,056 parameters (three layers of 128 neurons with residual connections and SiLU; inputs: the 64 pixels, 16 numbers encoding the step, and the label in 11 slots, the last one meaning “no hint”) and was trained for 30,000 rounds with batches of 256 images: 7,680,000 noisy versions of lesson 15’s 1,500 training digits. With the seven at step 20, its noise estimate has a mean squared error of 0.135 per pixel; if it said “there is no noise,” the error would be 0.864.

Sampling

DDPM’s original sampler starts from $x_T\sim\mathcal N(0,I)$ and, for $t=T,\dots,1$, computes

with a little fresh noise $z$ at each step (except the last). The scene uses the deterministic variant of Song et al. (2021), DDIM with $\eta = 0$, with the same network trained as in DDPM:

That is: the noise estimate gives the clean image it implies, which is then mixed again with the share of noise that belongs to the previous step. With no fresh noise, each point on the plane follows its arrow exactly and the same static always gives the same result; different static gives a different one.

Classifier-free guidance

A single network is trained with the label $c$ and, with probability 0.15, without it ($\varnothing$). When sampling, they are combined:

With $s=0$ it is the estimate without the hint (the hint is ignored); with $s=1$, the conditional, which follows the hint as is; with $s>1$ the hint’s direction is exaggerated. It is the same convention as in lesson 18. Ho and Salimans (2022) write the same with $w = s-1$: $\tilde\varepsilon = (1+w)\,\varepsilon_\theta(x_t,c) - w\,\varepsilon_\theta(x_t)$. Each guided step costs two network evaluations. On the lesson’s plane, with the hint “seven” and two hundred points (reader: lesson 15’s logistic regression), 26 are read as sevens with $s = 0$, 199 with $s = 1$, and 200 with $s = 5$. At the same time, the real sevens left with no new point within 0.3 number 4 of 154 with $s = 1$, 48 with $s = 2$, 95 with $s = 3$, and 144 with $s = 5$: the points crowd onto the edge of the group farthest from the other digits. The pixel network generates its sevens with $s = 3$.

Latent diffusion

An autoencoder compresses the image into a code $z=E(x)$; diffusion happens in $z$ and at the end $x=D(z)$ is decoded (Rombach et al., 2022). In the lesson, $z$ is the two numbers of lesson 15’s VAE, multiplied by 1.6 before diffusing (Stable Diffusion also rescales its codes by a fixed factor), and the plane network has 10,370 parameters. In Stable Diffusion v1, the autoencoder shrinks each side 8 times and outputs 4 channels: a $512\times512\times3 = 786{,}432$-number image becomes a $64\times64\times4 = 16{,}384$-number code, 48 times fewer.

In code

import numpy as np

T, s = 50, 0.008
f = lambda t: np.cos((t / T + s) / (1 + s) * np.pi / 2) ** 2
a_full = f(np.arange(T + 1)) / f(0)
betas = np.clip(1 - a_full[1:] / a_full[:-1], 1e-4, 0.35)   # noise schedule
alphas = 1 - betas
abar = np.cumprod(alphas)
print('image share of the mix (%):', [round(float(100 * abar[t - 1]), 1) for t in (10, 25, 40, 50)])

# Toy data: a single number that is -1 or +1. For it, the best possible
# noise estimator can be written by hand (in the lesson a network learns it).
def estimated_noise(x, i):
    a, b = np.sqrt(abar[i]), np.sqrt(1 - abar[i])
    x0 = np.tanh(a * x / b**2)          # mean of x0 given x_t
    return (x - a * x0) / b

rng = np.random.default_rng(0)
x = rng.standard_normal(10_000)        # pure static
for i in reversed(range(T)):           # fifty steps back (DDPM)
    e = estimated_noise(x, i)
    x = (x - betas[i] / np.sqrt(1 - abar[i]) * e) / np.sqrt(alphas[i])
    if i > 0:
        x += np.sqrt(betas[i]) * rng.standard_normal(x.size)

print('within 0.1 of -1 or +1:', round(float(np.mean(np.abs(np.abs(x) - 1) < 0.1)), 3))
print('share at +1:', round(float(np.mean(x > 0)), 3))
# image share of the mix (%): [89.9, 49.4, 9.4, 0.3]
# within 0.1 of -1 or +1: 1.0
# share at +1: 0.499

What comes next

Lesson 18 shows how a sentence becomes the hint that guides diffusion (CLIP and text conditioning). Later on, the transformer (lessons 21 and 22) replaces the convolutional network in many modern diffusion models.