Scaling laws and costs

The power law and the logarithmic scale

Kaplan et al. (2020) trained transformers from 768 to 1.5 billion parameters (not counting embeddings) on WebText2 text and fitted the loss $L$ (cross-entropy, in nats per token) against the number of parameters $N$, with plenty of data:

Taking logarithms, $\log L=\alpha_N\log N_c-\alpha_N\log N$: a straight line with slope $-\alpha_N$ when both axes are on a logarithmic scale. Multiplying $N$ by $k$ multiplies $L$ by $k^{-\alpha_N}$: by 10, $10^{-0.076}\approx 0.84$. There are similar lines for data ($\alpha_D\approx 0.095$, that is ×0.80 per ×10) and for well-allocated compute ($\alpha_C\approx 0.050$, ×0.89). The constants depend on the vocabulary and the data, and outside the measured range everything is extrapolation. The lesson draws this published formula, not the raw measurements.

Why six: $C\approx 6ND$

On the way forward, each parameter takes part in one multiplication and one addition per token: about $2N$ FLOPs (Kaplan et al., eq. 2.2, ignoring the attention term, small when $d_{\text{model}}\gg n_{\text{ctx}}/12$). The way back computes two things, the gradient with respect to the weights and with respect to the activations, and each costs about as much as the forward pass (Brown et al., 2020, table D.1): about $4N$. In total, $6N$ per token and

Hoffmann et al. (2022, table A4) compared their exact count with $6ND$ for models from 73M to 6.8B parameters: the ratio was 0.99 to 1.10. For GPT-3, $6\cdot 175\cdot 10^{9}\cdot 300\cdot 10^{9}=3.15\cdot 10^{23}$; the paper reports $3.14\cdot 10^{23}$ with $N=174.6$B.

A fixed budget (Chinchilla)

Hoffmann et al. propose for the loss

$E$ is the loss that cannot be removed; the second term is what is missing from having a finite model; the third, from having seen finite data. Minimizing $\hat L$ with $6ND=C$ fixed gives $N_{\text{opt}}\propto C^{a}$ and $D_{\text{opt}}\propto C^{b}$ with $a\approx b\approx 0.5$: parameters and data grow together, and their table 3 gives about 20 tokens per parameter (1B → 20.2B tokens; 10B → 205B). Kaplan had found $N_{\text{opt}}\propto C^{0.73}$: with 10 times more compute, 5.5 times more parameters and only 1.8 times more data. Besiroglu et al. (2024) refitted the five constants on data reconstructed from the paper’s plots: the published ones did not agree with the paper’s other two methods, and theirs do (their point estimate gives about 26 tokens per parameter). Llama 3 ran its own fit and, for $3.8\cdot 10^{25}$ FLOPs, predicted a 402B model with 16.55 trillion tokens.

Twenty is a guide for spending training compute well, not a law. If the model will be used a lot, a smaller one trained on more tokens pays off, because each answer costs less (Sardana et al., 2024): the 8B Llama 3.1 read over 15 trillion tokens.

From FLOPs to hours and money

With a peak speed $P$ per GPU and a utilization fraction (MFU, model FLOPs utilization, Chowdhery et al., 2022),

$H$ is GPU-hours and $t$ the time with $G$ GPUs at once. Since $H$ does not depend on $G$, doubling the GPUs halves the time and leaves the cost unchanged, as long as utilization holds. The H100 datasheet gives 1,979 teraFLOPS in BF16 “with sparsity”; without it, it is half (NVIDIA, 2021: 2:4 sparsity doubles peak throughput), about 989. Llama 3 405B measured 380 to 430 teraFLOPS per GPU, an MFU of 38 to 43% (table 4), with up to 16,384 GPUs and 466 interruptions over a 54-day period. PaLM reports 46.2% and estimates 21.3% for GPT-3.

In code

def training(n, d, useful=400e12, price=2.0, gpus=16384):
    flops = 6 * n * d                  # C ≈ 6·N·D
    hours = flops / useful / 3600      # GPU-hours
    return flops, hours, hours / gpus / 24, hours * price

models = [("Pythia 1.4B", 1.4e9, 300e9), ("GPT-3", 175e9, 300e9),
          ("Chinchilla", 70e9, 1.4e12), ("Llama 3.1 405B", 405e9, 15.6e12)]
for name, n, d in models:
    c, h, days, usd = training(n, d)
    print(f"{name:15} C = {c:.2e}  hours = {h:9.0f}  days = {days:5.1f}  USD {usd:9.0f}")

print(round(10 ** -0.076, 3))          # ×10 in N: the loss is multiplied by this

Output:

Pythia 1.4B     C = 2.52e+21  hours =      1750  days =   0.0  USD      3500
GPT-3           C = 3.15e+23  hours =    218750  days =   0.6  USD    437500
Chinchilla      C = 5.88e+23  hours =    408333  days =   1.0  USD    816667
Llama 3.1 405B  C = 3.79e+25  hours =  26325000  days =  66.9  USD  52650000
0.839

The 400 trillion useful operations per second, the 2 dollars per hour, and the 16,384 GPUs are assumptions (the first two are neither prices nor measurements for each model; GPT-3 and Chinchilla were trained on other hardware). The figure covers only the final run: no experiments, failures, data, or people. Meta reported 30.84 million H100 GPU-hours for Llama 3.1 405B.

What comes next

Lesson 25 counts bytes instead of operations: which model fits in a GPU’s memory and how the work is split. Lesson 26 measures the cost of using the model, which with heavy demand outweighs the cost of training it. Lesson 27 reuses a base model instead of repeating this arithmetic.