The deep learning big bang

The 2012 numbers

From the AlexNet paper (Krizhevsky, Sutskever and Hinton, 2012): eight layers with weights (five convolutional and three dense), about 60 million parameters, about 650,000 neurons, more than 1.2 million training images, and five to six days on two GTX 580 3 GB GPUs. The input has 150,528 values ($224\times224\times3$). Two more details from the same paper: in its figure 1, a four-layer network on CIFAR-10 reached 25% training error six times faster with ReLU than with tanh (the authors note the effect varies with the architecture), and dropout, with probability 0.5 in the first two dense layers, was used because without it the network overfit substantially, at the cost of roughly doubling the iterations needed.

Model in ILSVRC 2012	Top-5 error
SIFT + Fisher vectors (runner-up)	26.2%
1 convolutional network (validation)	18.2%
5 networks averaged	16.4%
7 networks, 2 of them pre-trained on 15 M images	15.3%

So the “15.3%” figure comes from an ensemble of seven networks and extra data; with only the contest data it was 16.4%. Even so, the gap to the runner-up is huge.

Multiplying matrices: why the GPU wins

The product $C=AB$ of two $n\times n$ matrices is

That is $n^2$ independent cells, each costing $n$ multiply-adds: $n^3$ in total, or $2n^3$ floating-point operations (FLOP: one addition or multiplication with decimals). With $P$ cores doing one multiply-add per step, the ideal number of steps is

With $P=512$ (GTX 580) the ideal ceiling on the advantage is 64; with $P=3584$ (RTX 3060), 448. It is an idealized count: it ignores memory bandwidth, the cost of moving data to the GPU, and the fact that each GPU core is slower than a CPU core. The real measurements of Raina, Madhavan and Ng (2009), with a GTX 280 GPU against a dual-core CPU, were between 12.9 and 72.6 times faster, depending on the model size.

A chip’s peak speed is $\text{cores}\times 2\times\text{clock}$: $512\times2\times1.544\ \text{GHz}\approx1.58$ TFLOP/s for the GTX 580 (the figure OpenAI uses) and, by the same arithmetic, about 12.7 TFLOP/s for the RTX 3060.

The cost of training AlexNet

According to the estimate by Amodei and Hernandez (OpenAI, 2018), the forward pass costs about $0.75\times10^{9}$ multiply-adds per image; training adds 2 operations per multiply-add, the backward pass (a factor of 3), and about 90 passes over the dataset:

That is consistent with two GTX 580s for 5.5 days at 33% of their peak, which is what OpenAI assumes: $2\cdot1.58\times10^{12}\cdot5.5\cdot86{,}400\cdot0.33\approx5\times10^{17}$. Since 2012 (from AlexNet to AlphaGo Zero, in late 2017), OpenAI measured the compute of the largest training runs doubling every 3.4 months; later estimates (Sevilla et al., 2022) give about 6 months for the deep learning era. Either way, growth is exponential.

The gradient through the layers

In a plain stack, $h_l=\tanh(W_l h_{l-1})$ and the gradient flows back as

so its norm is multiplied, at each layer, by a factor of roughly $\sigma\cdot\overline{\tanh'}$. With the era’s “standard” initialization, $n\,\mathrm{Var}[W]=1/3$ (Glorot and Bengio, 2010, eq. 15), the factor is about 0.5; in the lesson’s toy it comes out ≈ 0.55, and $0.55^{40}\approx 4\times10^{-11}$.

With a shortcut, $x_{l+1}=x_l+F(x_l)$ and, according to He et al. (2016, eq. 5),

The $I$ term carries the gradient from the output to any layer without passing through any weight. In the toy, each block also adds its contribution and the norm grows a little (between 2 and 18 at 40 layers, depending on the seed).

What the toy does not say

The toy network is untrained, uses no normalization, and all its layers are alike. In real networks, initialization and batch normalization keep signals healthy. In fact, He et al. (2015, §4.1 and figure 1) argue that the difficulty of training deep plain networks does not seem to come from vanishing gradients: their plain networks had gradients of healthy norm and still the 34-layer one had higher training error than the 18-layer one, and the 56-layer one more than the 20-layer one on CIFAR-10. They called it the degradation problem. Their hypothesis is that it is easier for a block to learn $F\approx0$ (identity) than to fit an identity with nonlinear layers; the authors themselves leave that explanation as an open question. What was measured is clear: with shortcuts, networks of 50, 101, and 152 layers improved with depth. ResNet-152 (11.3 billion multiply-adds) costs less than VGG-19 (19.6 billion); on its own it reached 4.49% top-5 error on validation, and an ensemble of six models, 3.57% on the test set.

In code

import numpy as np

def grad_norms(depth, residual, width=16, seed=0):
    rng = np.random.default_rng(seed)
    W = rng.normal(0, np.sqrt(1/3) / np.sqrt(width), size=(depth, width, width))
    h = rng.normal(size=width)
    cache = []
    for l in range(depth):                        # forward pass
        t = np.tanh(W[l] @ h)
        cache.append(t)
        h = h + t if residual else t
    g = np.ones(width) / np.sqrt(width)           # gradient of norm 1 at the output
    norms = [np.linalg.norm(g)]
    for l in reversed(range(depth)):              # backpropagation
        branch = W[l].T @ (g * (1 - cache[l] ** 2))
        g = g + branch if residual else branch    # with a shortcut, the gradient passes untouched and the branch is added
        norms.append(np.linalg.norm(g))
    return norms                                  # norms[k]: norm after k layers backward

for res in (False, True):
    n = grad_norms(40, res)
    print("with shortcuts" if res else "plain", f"after 10 layers: {n[10]:.2e}   after 40: {n[40]:.2e}")
# plain after 10 layers: 5.30e-03   after 40: 1.16e-10
# with shortcuts after 10 layers: 2.51e+00   after 40: 1.48e+01

The exact numbers depend on the seed and the random generator (the lesson’s gradients.py file also checks backpropagation against finite differences); the order of magnitude does not. And the ideal race count:

from math import ceil

def steps(n, cores):            # n² independent cells; each costs n multiply-adds
    return ceil(n * n / cores) * n

print(steps(64, 8), steps(64, 512), steps(64, 3584))   # 8-core CPU, GTX 580, RTX 3060
# 32768 512 128

What comes next

Residual connections and normalization reappear in every block of the transformer (lesson 22). Lesson 14 moves from seeing to representing meaning with embeddings. The accounting of FLOPs, data, and cost is formalized in lesson 24, and GPU memory, which already limited AlexNet, rules in lesson 25.