GANs: the forger and the detective

The game in one formula

The generator $G$ takes noise $z \sim p_z$ (in the lesson, the standard bell) and produces $G(z)$. The discriminator $D$ returns $D(x) \in (0, 1)$, the probability it assigns to $x$ coming from the data. Goodfellow et al. (2014) set up a two-player minimax game:

$D$ raises $V$ by saying “1” to real data and “0” to fakes (it is the cross-entropy of a binary classifier, with the sign flipped); $G$ lowers it by pushing $D(G(z))$ toward 1. What one gains, the other loses: it is a zero-sum game.

The best possible detective, and the tie

With $G$ fixed, the optimal discriminator is

where $p_g$ is the distribution of the fakes. If a region holds two fakes for every real point, the best answer there is $1/3$. That is what the collapsing match measures: on the left mountain (all the fakes and half the real data) the detective answers between 0.32 and 0.35; on the right, where there are no fakes, 1.00. The paper proves the game’s global optimum is at $p_g = p_{\text{data}}$, with $D = \tfrac12$ everywhere. There the detective’s loss equals $2\ln 2 \approx 1.386$ and the generator’s (non-saturating version, below) $\ln 2 \approx 0.693$; over the last two hundred steps of the lesson’s healthy match they average 1.384 and 0.702.

The non-saturating trick and the $k$ steps

Early on, $D$ rejects fakes with high confidence and $\log(1 - D(G(z)))$ saturates: it gives almost no gradient. So the same paper proposes that $G$ maximize $\log D(G(z))$ instead, which has the same fixed point and stronger gradients early on. The lesson uses it. The gradient that corrects $G$ flows through $D$ (backpropagation through both networks, lesson 8): for each fake, $\partial \log D / \partial x$ says which way to move it. The arrows in the scene are that derivative, computed from the detective’s real curve.

Algorithm 1 alternates $k$ steps of $D$ with one of $G$; the authors used $k = 1$. In the lesson, $k$ is the only hyperparameter that changes: with $k = 1$, all 20 seeds cover both mountains; with $k = 5$, 15 cover them, 4 collapse (one mountain gets under 5% of the fakes), and one ends up halfway (48% and 6%). The paper warns about the opposite imbalance, training $G$ too much without updating $D$, which it calls “the Helvetica scenario”: $G$ sends too many values of $z$ to the same $x$. Either way, the balance between the players matters.

The lesson’s GAN

• Data: a 50/50 mix of $\mathcal N(-1.5,\ 0.4^2)$ and $\mathcal N(1.5,\ 0.4^2)$. The scene’s 800 real points are quantiles of each bell, and the generator’s 800 draws are quantiles of $\mathcal N(0, 1)$, so every histogram is reproducible.
• Networks: $G$ and $D$ are identical, $1 \to 10\ \tanh \to 10\ \tanh \to 1$ (141 parameters each). Adam with rate $5 \times 10^{-4}$ and $\beta_1 = 0.5$, batches of 256, 2,000 generator steps. The weights were exported quantized to 16 bits and the browser runs the networks (forward pass only); the scripts train_gan.py and train_detective.py are in the lesson folder.
• The detective alone: to present it separately, the healthy match’s initial detective was trained for 1,000 steps against the frozen initial forger (everything near 0). It answers 0.95 to 1.6 and 0.05 to 0.07.
• “Near a mountain”: within 0.8 of its center. Real data: 47.75% in each. Fakes at the end of the healthy match: 46.4% and 48.1%.
• What did not come out perfect: the fake mountains are narrower (the tallest packs 337 of the 800 fakes into one 0.25-wide stretch; the real one, 98), and a bridge of fakes remains in the valley. The bridge is unavoidable with this design: $G$ is a continuous function of a single number, so the $z$ values near 0 land between the two mountains. There $D(0) \approx 0.30$. The detective, small and trained at the same pace, does not detect the narrowness.

In code

import torch, torch.nn as nn
torch.manual_seed(0)

def net():   # 1 → 10 → 10 → 1, with tanh: 141 parameters
    return nn.Sequential(nn.Linear(1, 10), nn.Tanh(), nn.Linear(10, 10), nn.Tanh(), nn.Linear(10, 1))

def real(n):   # 50/50 mix of N(−1.5, 0.4²) and N(+1.5, 0.4²)
    sign = torch.randint(0, 2, (n, 1)) * 2 - 1
    return 1.5 * sign + 0.4 * torch.randn(n, 1)

G, D = net(), net()               # forger and detective
opt_G = torch.optim.Adam(G.parameters(), lr=5e-4, betas=(0.5, 0.999))
opt_D = torch.optim.Adam(D.parameters(), lr=5e-4, betas=(0.5, 0.999))
bce = nn.BCEWithLogitsLoss()      # applies the sigmoid and computes −log
one, zero = torch.ones(256, 1), torch.zeros(256, 1)

for step in range(2000):
    # detective’s turn: it should say 1 to real data and 0 to fakes
    fakes = G(torch.randn(256, 1)).detach()
    loss_D = bce(D(real(256)), one) + bce(D(fakes), zero)
    opt_D.zero_grad(); loss_D.backward(); opt_D.step()
    # forger’s turn: make the detective say “real” to its fakes
    loss_G = bce(D(G(torch.randn(256, 1))), one)
    opt_G.zero_grad(); loss_G.backward(); opt_G.step()

with torch.no_grad():
    x = G(torch.randn(10000, 1))
    left = ((x + 1.5).abs() < 0.8).float().mean().item()
    right = ((x - 1.5).abs() < 0.8).float().mean().item()
    print(f'fakes near each mountain: {left:.2f} and {right:.2f}')
    dr, df = torch.sigmoid(D(real(10000))).mean().item(), torch.sigmoid(D(x)).mean().item()
    print(f'detective, on average: real {dr:.2f}, fakes {df:.2f}')
# fakes near each mountain: 0.47 and 0.48
# detective, on average: real 0.50, fakes 0.50

(PyTorch 2.11 on CPU; another version may give slightly different figures.)

What comes next

GANs dominated high-quality image generation until Dhariwal and Nichol (2021) showed that diffusion models beat them. They themselves point out that GANs capture less diversity and are hard to train, “collapsing without carefully selected hyperparameters and regularizers,” while diffusion offers distribution coverage, a stationary training objective (it does not depend on a changing rival), and easy scalability; its drawback is slower generation, in many noise-removal steps. That is lesson 17.