From text to image

CLIP’s contrastive loss

In a batch of $N$ pairs, the image encoder $f$ and the text encoder $g$ produce unit vectors $u_i = f(x_i)$ and $v_j = g(t_j)$. Their cosine similarity is $S_{ij} = u_i \cdot v_j$. These are divided by a temperature $\tau$ (also learned), and cross-entropy asks each image to pick its own sentence and each sentence to pick its own image:

This is the symmetric loss from Radford and colleagues’ paper (Figure 3): it raises the diagonal of the table $S$ and lowers the rest of its row and column. CLIP used batches of 32,768 pairs, that is, tables of more than a billion cells at every step, and about 400 million image-and-text pairs gathered from publicly available sources on the internet. The model card is dated January 2021.

In our toy, the image encoder receives 720 numbers (12 × 12 pixels, each with 5 one-hot color slots) and has a hidden layer of 96; the text encoder reads 5 word positions, each word with a 16-number vector plus a vector for its position, and has a hidden layer of 64. Both output $D = 32$. It trained for 2,500 steps with batches of 96; the temperature went from $1/\tau \approx 14.3$ to $22.0$. In the fixed six-pair batch the scene shows, the diagonal ends between 0.78 and 0.88. The batch error does not reach zero with batches of 96: many incomplete sentences (“circle”) repeat within the same batch and cannot tell their images apart.

Conditioning and guiding

The diffusion network predicts the noise $\epsilon_\theta(z_t, t, c)$ of a 16-number latent $z_t$, at step $t$, given the sentence vector $c$. During training, the sentence is replaced with the empty sentence $\varnothing$ 10% of the time (as in Stable Diffusion v1-4). When sampling, with guidance scale $s$:

With $s = 0$ the sentence does not count; with $s = 1$ it is the plain conditional network; with $s > 1$ it is exaggerated. This is the convention of the diffusers library (default scale 7.5, “enabled when greater than 1”). Ho and Salimans write the same thing as $(1+w)\,\epsilon_c - w\,\epsilon_\varnothing$, with $w = s - 1$, and present it as a trade-off between sample quality and diversity. GLIDE (Nichol and colleagues, 2021) already used it with text, and human evaluators preferred it to CLIP guidance.

Our sampler follows the lesson 17 recipe: $T = 40$ steps, $\beta_t$ linear from 0.02 to 0.30, and reverse-step variance $\sigma_t^2 = \beta_t$. The latent comes from a 16-number VAE and is decoded at the end. Stable Diffusion conditions differently (cross-attention over the per-word vectors from CLIP’s text encoder); here, to fit in the browser, the sentence vector is appended to the input of a three-layer network.

What the scene measured

• Guidance scale with “circle” (256 starting noises): circles 36% at $s = 0$, 80% at $s = 1$, 95% at $s = 2$, 93% at $s = 4$, and 63% at $s = 8$. The scene’s twenty-four images (seeds 1000 to 1023, prompt “circle”) give 11, 20, 23, and 13 circles at $s = 0, 1, 3, 8$, and 18, 11, 6, and 8 distinct combinations of shape, color, and background (with the Spanish prompt «círculo»: 11, 19, 21, and 16 circles; 18, 12, 7, and 7 combinations).
• Bias: in the data, 64.9% of circles are magenta (one image in two gets its shape’s typical color, the other a random one). Among the twenty-four images, 15 are magenta at $s = 1$ and 22 at $s = 3$.
• The never-seen (ten combinations left out of training, 16 noises per combination, $s = 3$): 86.4% correct on the 70 seen combinations and 75.6% on the 10 never-seen ones. At $s = 1$ the gap is wider: 85.6% versus 45.6%.
• What goes with what (60 pairs with swapped colors, in both languages): the encoder that reads word order picks the right sentence 100% of the time; the bag of words gives exactly the same vector to both sentences (0%: always a tie).

The shape and color counts come from a fixed-rule reader, not a learned one: the background is the most common color along the border, the shape is the rest, and the shape type is the template (circle, square, triangle, or cross, at several positions and sizes) that overlaps best; if none exceeds 0.6 intersection over union, the image counts as “no shape.” It reads all 20 real images in the lesson correctly. It is less reliable with squares: the model often draws them with rounded corners and the reader takes them for circles, so the scene does not use them for counting.

In code

import numpy as np

def contrastive_loss(U, V, tau=0.07):
    """U, V: unit vectors of N images and N sentences (row i = pair i)."""
    S = U @ V.T / tau
    S = S - S.max(axis=1, keepdims=True)
    rows = -np.mean(np.diag(S) - np.log(np.exp(S).sum(axis=1)))
    S2 = (U @ V.T / tau).T
    S2 = S2 - S2.max(axis=1, keepdims=True)
    cols = -np.mean(np.diag(S2) - np.log(np.exp(S2).sum(axis=1)))
    return (rows + cols) / 2

rng = np.random.default_rng(0)
U = rng.normal(size=(4, 8)); U /= np.linalg.norm(U, axis=1, keepdims=True)
V_random = rng.normal(size=(4, 8)); V_random /= np.linalg.norm(V_random, axis=1, keepdims=True)
V_good = U + 0.1 * rng.normal(size=(4, 8)); V_good /= np.linalg.norm(V_good, axis=1, keepdims=True)
print(round(contrastive_loss(U, V_random), 3), round(contrastive_loss(U, V_good), 6))

def guidance(e_without, e_with, s):
    return e_without + s * (e_with - e_without)

e_without, e_with = np.array([0.2, -0.4]), np.array([0.5, 0.1])
for s in (0, 1, 3):
    print(s, guidance(e_without, e_with, s))
# 6.064 0.000141
# 0 [ 0.2 -0.4]
# 1 [0.5 0.1]
# 3 [1.1 1.1]

With random sentences the error is high; with sentences almost identical to their image, the diagonal dominates and the error falls to nearly zero. The second function is the whole guidance scale.

What comes next

Here text only went in: it was turned into a vector. In lesson 19 text comes out: a language model assigns probabilities to the next word and writes one word after another.