The world as numbers

From image to vector

A grayscale image is a matrix $X \in \{0,\dots,255\}^{H \times W}$. Flattening it, row by row, gives the vector $\mathbf{x} \in \mathbb{R}^{HW}$ with

The little face in the lesson has $H = W = 8$, so $\mathbf{x}$ has 64 components. Each value fits in one byte: $2^8 = 256$ levels. Before training it is common to normalize, $x' = x/255 \in [0,1]$.

Neighborhood is lost when flattening: pixel $(i,j)$ and pixel $(i+1,j)$ are neighbors in the image, but their positions in the vector differ by $W$ (eight, for the little face). A model that treats each component as an independent measure, like the email model of lesson 1 or a fully connected network, does not know which pixels are neighbors; convolutional networks (lesson 11) do exploit it and receive the image as a matrix.

Color, channels, and tensors

A color image is a tensor $X \in \{0,\dots,255\}^{H \times W \times 3}$ with $3HW$ values. A 12-megapixel photo, for example $4000 \times 3000$ pixels, has

that is, about 36 MB uncompressed at one byte per value. Files like JPEG are much smaller, but the model works with the decoded pixels (or with a downsized version: AlexNet, for example, downsized every ImageNet image to $256 \times 256$ —rescaling the short side to 256 and cropping the center—, that is $256 \times 256 \times 3 = 196{,}608$ values, about 183 times fewer than the 12-megapixel photo; the network also took $224 \times 224 \times 3 = 150{,}528$-value crops; Krizhevsky et al., 2012). The order of the axes depends on the library: height × width × channels, or channels × height × width. Strictly, a number, a vector, and a matrix are also tensors (with 0, 1, and 2 axes).

In the lesson, the gray of each pixel comes from its color with the luma recipe of Recommendation ITU-R BT.601 (version 7, 2011): $Y' = 0.299\,R' + 0.587\,G' + 0.114\,B'$, applied to gamma-corrected values. With three 8-bit channels there are $256^3 = 16{,}777{,}216$ possible colors.

Blur: mixing with the neighbors

The lesson’s blur mixes each pixel with the average of its $3\times 3$ window, in a proportion $\lambda$:

with $\lambda = 0,\ 0.25,\ 0.5,\ 0.75,\ 1$ at levels 0 to 4, and $V_{ij}$ the window restricted to the neighbors that exist (9 pixels in the interior, 6 on the borders, 4 at the corners). For the pixel in row 3, column 3 (counting from 1), the nine values are 146, 202, 255, 202, 0, 202, 202, 0, and 202, and their average is $\approx 156.8$. It is a special case of convolution (lesson 11). Measured on the little face (values rounded to integers): at each level from 1 to 4 all 64 numbers change, with a mean change of 15.0, 29.9, 44.9, and 59.9 gray levels (out of 255), and the contrast (maximum minus minimum) drops from 255 to 205, 175, 145, and 131.

With $\lambda < 1$ the operation can be undone in theory (the matrix is invertible; at $\lambda = 0.75$ it already amplifies rounding 40 times), but with $\lambda = 1$ it really loses information. The sum over a window is $K \otimes K$, with $K$ the $8\times 8$ tridiagonal matrix of ones, and $K$ has a null vector, $u = (1,-1,0,1,-1,0,1,-1)$: three consecutive neighbors always sum to zero. So the full blur has rank $7 \times 7 = 49$, not 64, and 15 directions vanish. Exact example: adding $+25,-25,0,+25,-25,0,+25,-25$ to column 4 of the little face changes six pixels and gives another image with the same full blur, number by number.

Sound: sampling

A signal $s(t)$ is stored as $x[n] = s(n/f_s)$, with $f_s$ samples per second. Shannon (1949, theorem 1) states it like this: if a function contains no frequencies higher than $W$, it is completely determined by its values at points spaced $1/2W$ seconds apart; in practice one asks for $f_s > 2B$ for a signal with no frequencies above $B$ (at the exact limit, $f_s = 2B$, the tone of frequency $B$ is not determined, as the demonstration shows). Shannon credits Nyquist with the importance of that interval. Whisper resamples all audio to $f_s = 16{,}000$ Hz (Radford et al., 2022, section 2.2), which represents components up to 8 kHz: one minute is $16{,}000 \times 60 = 960{,}000$ values. The model does not receive those values as they are: it computes an 80-channel log-magnitude spectrogram with 25 ms windows and a 10 ms stride, that is 100 columns per second, $80 \times 100 = 8{,}000$ numbers per second.

The lesson’s toy signal adds a tone of 200 and one of 500 cycles per second and repeats every 10 ms; $n = f_s/100$ samples are taken per period. The dashed line is what an ideal converter would rebuild (trigonometric interpolation of the $n$ samples). It was checked with the lesson’s code: with $f_s \ge 1{,}100$ the reconstruction matches the signal with no error; with $f_s = 1{,}000$ the relative error is 0.43 (the 500 tone sits right at the limit) and below that it is larger still (0.67 to 1.65). That is why the gate asks you to lower it to 1,000 or less.

Text: from codes to vectors

Unicode assigns a number, the code point, to each character: “N” is 78, “e” is 101, “u” is 117, “ñ” is 241 (Unicode Standard code charts). It is a label, not a meaning. Language models split text into tokens (lesson 20), each token is an integer, and an embedding table $E \in \mathbb{R}^{V \times d}$ gives each one a vector $E_{\text{id}} \in \mathbb{R}^{d}$, learned during training. In GPT-3 175B the model dimension is $d_{\text{model}} = 12{,}288$ (Brown et al., 2020, table 2.1). The four features in the lesson ($d=4$) are made up; in a real model the features are learned and have no names. Closeness in the space, measured for example with the Euclidean distance $\lVert \mathbf{a}-\mathbf{b}\rVert$, gets filled with meaning in lesson 14. In the lesson, “cat” and “dog” are 0.22 apart; “cat” and “tiger”, 1.08; “cat” and “bread”, 1.68.

MNIST (LeCun et al., 1998) starts from digits written by US Census Bureau employees and by high-school students (NIST databases SD-3 and SD-1). The original images were black and white; normalizing them into a $20\times 20$ pixel box introduced the grays, and they were centered in a $28\times 28$ field by center of mass. The image in the lesson is an illustration, not a digit from the set.

In code

import numpy as np

# the little face in grayscale (8x8), the same one from the lesson
face = np.array([
    [ 20,  20, 146, 146, 146, 146,  20,  20],
    [ 20, 146, 202, 255, 202, 202, 146,  20],
    [146, 202,   0, 202, 202,   0, 202, 146],
    [146, 202,   0, 202, 202,   0, 202, 146],
    [146, 202, 202, 202, 202, 202, 202, 146],
    [146, 202,   0, 202, 202,   0, 202, 146],
    [ 20, 146, 202,   0,   0, 202, 146,  20],
    [ 20,  20, 146, 146, 146, 146,  20,  20],
], dtype=np.uint8)

vector = face.flatten()                 # from table to list, row by row
print(face.shape, vector.shape)         # (8, 8) (64,)

# a pixel and the one below it: neighbors in the image, far apart in the vector
i, j = 2, 2
print(i * 8 + j, (i + 1) * 8 + j)       # 18 26

# blur of one pixel: average of its 3x3 window
window = face[i - 1:i + 2, j - 1:j + 2]
print(window.mean())                    # 156.77777777777777

# blur of the whole image (at the borders, only the neighbors that exist)
def blur(img):
    out = np.zeros(img.shape)
    for r in range(8):
        for c in range(8):
            out[r, c] = img[max(r - 1, 0):r + 2, max(c - 1, 0):c + 2].mean()
    return out

x = face.astype(float)
other = x.copy()
other[:, 3] += [25, -25, 0, 25, -25, 0, 25, -25]   # another image: six different pixels
print(int((other != x).sum()), np.abs(blur(x) - blur(other)).max())   # 6 0.0

# light blur (25 %): every number changes, but only a little
light = 0.75 * x + 0.25 * blur(x)
print(round(np.abs(light - x).mean(), 1), int((np.round(light) != x).sum()))    # 15.0 64

# a 12-megapixel color photo (4000 x 3000 pixels, 3 channels)
photo = np.zeros((3000, 4000, 3), dtype=np.uint8)
print(photo.shape, photo.size)          # (3000, 4000, 3) 36000000

# text: the code of each letter
print([ord(c) for c in "Neu"])          # [78, 101, 117]

What comes next

With the data in vectors, lesson 3 puts the first model on top: a line $y = wx + b$ and its error. Later, convolutions (lesson 11) recover pixel neighborhood, embeddings (lesson 14) give meaning a geometry, and tokens (lesson 20) explain which text “pieces” a language model really reads.