Pretraining

The objective, in one formula

For a text of $T$ pieces $x_1,\dots,x_T$, the model with parameters $\theta$ gives, at each place, a probability to the piece that really comes next. The error of a bet is $-\ln p$, and pretraining minimizes its average, the cross-entropy:

With the causal mask from lesson 22, all $T-1$ predictions of a text are computed in one pass. If the right piece gets $p=1$, the error is $0$; as $p\to 0$, it grows without limit. Guessing at random among the $V = 50{,}254$ pieces of Pythia’s tokenizer gives $\ln V \approx 10.82$ (nats per piece). The scene’s numbers: untrained, “five” gets $p = 3.28\cdot10^{-5}$, error $10.33$; after training, $p = 0.1175$, error $2.14$.

How we measured Pythia

Pythia (Biderman et al., 2023) is a family of 16 open models trained on the same data, in the same order, with 154 public checkpoints per model: step 0, steps 1, 2, 4, …, 512, and every 1,000 steps up to 143,000. Each step uses 1,024 sequences of 2,048 tokens ($2{,}097{,}152$ tokens), so the total is $299{,}892{,}736{,}000 \approx 300$ billion. We used the 1.4-billion-parameter model (1,414,647,808 in total), trained on The Pile without deduplication, at 25 of its checkpoints. The script build-pythia.py (next to this lesson) loads each one, measures, and writes pythia-data.ts; nothing is edited by hand. Everything is deterministic: greedy decoding, no sampling.

The curve’s “average error” is $\mathcal{L}$ on a 500-token English text written for this lesson in 2026 (The Pile is from 2020: it could not have read it). It is little text: good for seeing the shape of the curve, not for comparing models. Broken down by kind of piece, form clearly arrives before content:

Step (tokens read)	Punctuation	Frequent words	Other words	Names and capitals	Average
0 (0)	11.40	10.86	11.02	11.32	11.01
16 (34M)	2.94	7.34	11.09	11.32	8.65
128 (268M)	2.40	4.40	9.70	10.42	6.77
1,000 (2.1B)	1.46	2.82	5.94	7.64	4.21
16,000 (33.5B)	0.94	2.00	3.84	6.33	2.82
143,000 (300B)	0.80	1.84	3.39	6.14	2.52

Punctuation drops from 11.4 to 2.9 within the first 34 million tokens; proper names are still near 10 at 500 million and, at the end, remain the hardest (6.1). The answers: “Paris” first appears at step 16,000 (before that it says “the capital of France,” “the city of Lyon,” “Bordeaux”), with $p$ going from 0.30 to 0.61; “casa” at the same step, with $p$ going from 0.15 to 0.82. Counting (“1, 2, 3, 4, 5,” → “6”), on the other hand, already works at step 1,000: some simple patterns come early. In Spanish, “París” is never the most likely piece: at the end it gets $p = 6.7\cdot10^{-5}$. To compare languages regardless of the tokenizer one uses bits per byte: at the end, 0.78 in English and 0.99 on the same text translated into Spanish.

Near-duplicates

Each document becomes the set $A$ of its chunks of 5 words in a row (shingles). The similarity of two documents is the Jaccard index:

In the scene’s sentence, $|A\cup B| = 6$ and $|A\cap B| = 2$: $J = 0.33$. Comparing everything with everything is impossible with billions of pages, so MinHash is used (Broder, 1997): with a hash function $h$, the probability that the chunk with the lowest $h$ is the same in $A$ and in $B$ is exactly $J(A,B)$. With many hashes grouped into buckets, documents that match on a whole bucket are flagged as duplicates. FineWeb (Penedo et al., 2024) uses 5-grams and 112 hashes in 14 buckets of 8, targeting documents at least 75% similar; Pythia has a version trained on a deduplicated Pile, threshold 0.87, which went from about 300 to 207 billion tokens; GPT-3 shrank each dataset by 10% on average this way. Lee et al. (2022) found in C4 a 61-word sentence repeated more than 60,000 times, and with deduplicated data their models emitted memorized training text ten times less often.

Quality filters and the mix

The scene’s pipeline (crawl.ts) really applies three FineWeb rules: a page is dropped if at most 12% of its lines end in punctuation, if at least 10% of its characters are in repeated lines, or if at least 67% of its lines are shorter than 30 characters; and two from C4 (Raffel et al., 2020): “lorem ipsum” and curly brackets {. Emails and IP addresses are masked, as in the public release of FineWeb. GPT-3 filtered Common Crawl with a classifier that rewarded resemblance to high-quality reference texts: 45 TB compressed became 570 GB. The mix is not proportional to size: Wikipedia (3 billion tokens) weighs 3% and is read 3.4 times; filtered Common Crawl (410 billion) weighs 60% and is read 0.44 times. FineWeb-Edu, 1.3 trillion tokens selected for educational value, matches another web dataset on MMLU with almost 10 times fewer tokens.

In code

import math

def shingles(text, k=5):
    w = text.lower().split()
    return {' '.join(w[i:i + k]) for i in range(len(w) - k + 1)}

def jaccard(a, b):
    return len(a & b) / len(a | b)

A = shingles('plant two thousand trees along the river path')
B = shingles('plant three thousand trees along the river path')
print(len(A), len(A | B), len(A & B), round(jaccard(A, B), 2))

# error of a bet: −ln p of the right piece
for p in (3.279e-05, 0.1175):
    print(round(-math.log(p), 2))
print(round(math.log(50254), 2))   # guessing at random among 50,254 pieces
# 4 6 2 0.33
# 10.33
# 2.14
# 10.82

Rights, consent, and privacy

The Pile (Gao et al., 2020) marks in its table 5 which components have author consent; Books3 (books from a copy of the private site Bibliotik) does not. In Bartz v. Anthropic (Northern District of California), the order of June 23, 2025 found fair use in using the books for training and in digitizing purchased books, but not in downloading pirated copies for a permanent library; the $1.5 billion settlement, about $3,000 per work for some 482,000 works, was approved on July 20, 2026. It is one court’s decision in one country; other lawsuits are ongoing and other jurisdictions have different rules. Consent is shifting too: Longpre et al. (2024) estimate that in a single year (April 2023 to April 2024), a quarter or more of the tokens from C4’s most important sites became restricted for AI crawlers in their robots.txt. Carlini et al. (2021) extracted from GPT-2 604 memorized texts out of 1,800 candidates; among them were names, phone numbers, and emails, some present in a single training document.

What comes next

Pythia read 300 billion tokens with 1.4 billion parameters. Lesson 24 shows how the error falls as parameters, data, and compute grow, and what that costs. How a base model becomes an assistant is the subject of lessons 27 and 28.