Tokens: the model’s alphabet

BPE step by step

Take a corpus of words with frequencies $f(w)$ and an initial alphabet $V_0$. On each round we count how many times each pair of neighboring pieces $(a,b)$ appears, weighted by the frequency of the word:

We pick $(a^*,b^*)=\arg\max_{(a,b)}\operatorname{count}(a,b)$, replace that pair with a new piece throughout the corpus, and add it to the vocabulary: $V_{t+1}=V_t\cup\{a^*b^*\}$. After $M$ merges, $|V|=|V_0|+M$. To tokenize new text, the merges are applied in the order they were learned. The idea comes from a compression algorithm (Gage, 1994) that replaced the most frequent pair of bytes with an unused byte; Sennrich, Haddow and Birch (2016) adapted it to split rare words in machine translation, merging characters instead of bytes. It is not the only way to build a vocabulary, but it is the one used by GPT-2, Llama 2 and OpenAI’s tokenizers (tiktoken is a BPE tokenizer).

In the scene’s small example (cat ×5, cats ×3, hat ×4, hats ×2, catnip ×1), the pair “a t” adds up to $5+3+4+2+1=15$ and wins; then “c at” adds up to $9$; then hat (6), cats (3) and hats (2) are born. “Catnip” stays as cat · n · i · p because none of its remaining pairs appears twice.

Bytes: why nothing is left out

Tokenizers from GPT-2 onward start from the $256$ possible values of a byte. In UTF-8 encoding, an unaccented letter takes one byte, an accented letter such as “ó” or “ñ” takes two, and many emojis take four, so any text can be written with those $256$ pieces: whatever the vocabulary does not cover falls back to single bytes. That is why, in the Spanish version of the scene, the “ó” in Quetzalcóatl first appears as two pieces (bytes 195 and 179) joined by a dashed border. Before merging, GPT-2 splits the text into chunks (a word with its leading space, a number, some punctuation) and merges never cross from one chunk to another, so the vocabulary does not fill up with variants like “dog.”, “dog!” and “dog?”. That is why many pieces carry the space in front of them: “ cat” and “cat” are different tokens. The scene leaves that space out when drawing GPT-4o’s cuts, to keep them readable.

The lesson’s toy tokenizer is a byte-level BPE with that same pre-tokenization, trained on about seventeen hundred words written for the course (mostly English): $|V_0|=256$ and $M=400$, so $|V|=656$. GPT-2 uses $256$ bytes, $50{,}000$ merges and one special end-of-text symbol: $256+50{,}000+1=50{,}257$, the figure given in its paper.

Vocabulary size

Every piece needs a row in the embedding table $E\in\mathbb{R}^{|V|\times d}$, that is, $|V|\cdot d$ parameters, and the model’s final layer produces one score per piece. With $|V|=50{,}257$ and $d=768$ (the smallest GPT-2), the table has $50{,}257\times 768\approx 38.6$ million parameters. In exchange, a larger vocabulary compresses more: the Llama 3 report says its 128-thousand-piece vocabulary (versus Llama 2’s 32 thousand) raises characters per token from 3.17 to 3.94 on a sample of English.

Cost

With different per-million-token prices for input ($p_{\text{in}}$) and output ($p_{\text{out}}$):

For English, $n\approx \text{words}/0.75$. Example with an assumed price of 2 dollars per million: a 3,000-word report is about 4,000 tokens; a thousand reports, 4 million tokens and 8 dollars. In Anthropic’s price list (checked in October 2026), output tokens cost five times as much as input tokens for every model in the table.

Language, measured

With the 1,012 parallel English and Spanish sentences of the FLORES-200 dataset (devtest), counted with tiktoken 0.14.0:

tokenizer	English	Spanish	ratio
GPT-2 (gpt2)	27,044	53,578	1.98
GPT-4 (cl100k_base)	27,182	41,974	1.54
GPT-4o (o200k_base)	26,873	35,415	1.32

Part of the difference is that the Spanish translation uses more words (26,081 versus 21,901); the rest is that each Spanish word is cut into more pieces. English comes out at 0.81 words per token, close to the ¾ rule of thumb. Petrov et al. (2023) measured differences of up to fifteen times between languages for the same translated text. The four sentences in the “your turn” step were written by us: they illustrate, they are not a basis for budgeting.

Why letters and digits fail

The model receives identifiers, not characters: it must have learned from data how each piece is spelled. With GPT-4o’s tokenizer, “ strawberry” and “ carretera” (with their leading space, as they appear inside a question) are a single token each. Edman, Schmid and Fraser (2024) found that most models seem to know the spelling of their tokens but fail to use it well to manipulate text. With numbers, GPT-3.5 and GPT-4 have tokens for every one-, two- and three-digit number and cut from left to right; Singh and Strouse (2024) found that forcing a right-to-left cut (separating thousands with commas) greatly improves arithmetic. Llama 2, by contrast, splits all numbers into single digits.

In code

A complete BPE in twenty lines of Python, on the animation’s corpus:

from collections import Counter

def train_bpe(words, max_merges):
    """words: {word: frequency}. Returns the merges (a, b, count) and how each word ends up."""
    pieces = {w: list(w) for w in words}
    merges = []
    for _ in range(max_merges):
        counts = Counter()
        for w, p in pieces.items():
            for a, b in zip(p, p[1:]):
                counts[(a, b)] += words[w]              # weight by the word's frequency
        if not counts or max(counts.values()) < 2:
            break
        a, b = min(counts, key=lambda pair: (-counts[pair], pair))   # most frequent; ties: alphabetical
        merges.append((a, b, counts[(a, b)]))
        for p in pieces.values():
            i = 0
            while i < len(p) - 1:
                if p[i] == a and p[i + 1] == b:
                    p[i:i + 2] = [a + b]
                else:
                    i += 1
    return merges, pieces

corpus = {"cat": 5, "cats": 3, "hat": 4, "hats": 2, "catnip": 1}
merges, pieces = train_bpe(corpus, 10)
for a, b, n in merges:
    print(f"{a} + {b} -> {a + b}  ({n} times)")
print(pieces["cats"], pieces["catnip"])
# a + t -> at  (15 times)
# c + at -> cat  (9 times)
# h + at -> hat  (6 times)
# cat + s -> cats  (3 times)
# hat + s -> hats  (2 times)
# ['cats'] ['cat', 'n', 'i', 'p']

And with a real tokenizer (pip install tiktoken; output measured with version 0.14.0):

import tiktoken

enc = tiktoken.get_encoding("o200k_base")          # GPT-4o's tokenizer
for text in ["My cat is named Quetzalcoatl.", "How many r's are in strawberry?", "1234567"]:
    ids = enc.encode(text)
    print(len(ids), ids, [enc.decode([i]) for i in ids])
print(enc.n_vocab)
# 10 [5444, 9059, 382, 11484, 3106, 21156, 280, 1191, 69282, 13] ['My', ' cat', ' is', ' named', ' Qu', 'etz', 'al', 'co', 'atl', '.']
# 8 [5299, 1991, 428, 885, 553, 306, 101830, 30] ['How', ' many', ' r', "'s", ' are', ' in', ' strawberry', '?']
# 3 [7633, 19354, 22] ['123', '456', '7']
# 200019

What comes next

Each token number selects a row of the embedding table (lesson 14); from there on the model only sees vectors. In lesson 21 we will see how each token decides which others to look at (attention). The cost of generating token by token returns in lesson 26 (inference), and the size of the context, also measured in tokens, in lesson 30.