Hallucinations and limits

What training optimizes

Pretraining minimizes cross-entropy on real text:

The loss goes down when the model gives high probability to the piece that actually followed in the text. There is no “truth” variable in the formula: if the text contains an error, the objective rewards it all the same. Afterwards, tuning with human preferences (RLHF, lesson 28) adds another signal and can change calibration: the GPT-4 technical report (OpenAI, 2023) says the pretrained model was highly calibrated and that after post-training the calibration was reduced.

Facts seen only once

Kalai, Nachum, Vempala, and Zhang (2025) formalize the effect of rarity. Let $s_r$ be the fraction of training facts that appear exactly once (the “singleton rate,” from the Good–Turing missing-mass estimator). For arbitrary facts, with no pattern to generalize (like a birthday), a base model’s error satisfies, with probability at least 99%,

where $N$ is the number of examples, $E_c$ the set of plausible wrong answers for context $c$, and $\delta$ measures miscalibration. In the authors’ words: if 20% of birthday facts appear exactly once in the pretraining data, one expects base models to hallucinate on at least 20% of birthday facts. It is a theoretical result for facts with no structure; it does not predict the rate of a specific model. Kandpal et al. (2023) observe it in practice: accuracy on factual questions grows with the number of pretraining documents related to the question.

Why tuning does not remove it: exams reward guessing

Kalai et al. (2025) also explain why hallucinations survive post-training: most evaluations that dominate leaderboards grade in a binary way (1 if right, 0 if not), and “I don’t know” scores 0. On a four-option multiple-choice exam, guessing is worth $\tfrac14$ of a point on average and leaving it blank, 0: guessing always pays. They propose stating a confidence threshold $t$ in the instructions and penalizing each error with $t/(1-t)$ points, so that answering “I don’t know” is worth 0 and guessing stops paying when confidence is low.

Knowledge cutoff and what goes stale

Pythia was trained on The Pile (Gao et al., 2020), released on December 31, 2020: it cannot know anything later. Asked “Who won the 2022 FIFA World Cup?”, with the same five-example prompt and greedy decoding, Pythia-1.4B answers “Germany” with probability 0.46 (Argentina won, according to FIFA). It answers the same for 2014 (right) and for 2018 (wrong: France won). Old facts also linger: asked “What is the capital of Brazil?”, it answers “Rio de Janeiro” (0.53), which stopped being the capital on April 21, 1960, when Brasília was inaugurated (IBGE). Measurements: run-cutoff.py and run-qa.py in the lesson folder.

Calibration and expected calibration error

Split the $N$ answers into $B$ bins by confidence. With $n_b$ answers in bin $b$, its mean accuracy $\mathrm{acc}(b)$ and mean confidence $\mathrm{conf}(b)$:

(Guo et al., 2017). With ten equal-width bins, the lesson’s four hundred twenty-four questions give an ECE of ≈ 0.18 for Pythia-1.4B; the scene uses five bins so they read better. Almost all of the error is under-confidence: its answers are right more often than its probability says, partly because the probability of a string splits the mass among ways of writing the same thing (“Cervantes,” “Miguel de Cervantes”). Farquhar et al. (2024) group answers by meaning (semantic entropy) to correct that effect.

The lesson’s data are our own measurements, not a published benchmark: 424 general-knowledge questions written for the lesson (capitals, authors, dates, chemical symbols…), in English, with five solved examples in the prompt and greedy decoding. “Confidence” is the probability of the whole answer string. They are easy questions for a model trained on the internet: with truly rare facts, the error would be higher.

Answer or stay silent

If being right adds 1, being wrong subtracts $k$, and staying silent is worth 0, and the probability of being right is $p$, answering is worth $p - k(1-p)$ on average. It pays to answer when

which is Chow’s reject rule (1970). With $k = 1, 3, 9, 19$ the threshold is $0.5,\ 0.75,\ 0.9,\ 0.95$. The rule holds if $p$ is calibrated. With Pythia’s raw probability the best measured thresholds are lower, precisely because of its under-confidence: you must recalibrate (for example, with temperature scaling, Guo et al., 2017) or set the threshold directly on your own data.

TruthfulQA, in detail

Lin, Hilton, and Evans (2021) wrote 817 questions in 38 categories (health, law, finance, politics…) that some people would answer wrongly because of a false belief. The best model (175-billion-parameter GPT-3 with a “helpful” prompt) was truthful on 58% of the questions, but only 21% of its answers were both truthful and informative: “I have no comment” counts as truthful. People reached 94%. Within each model family, the largest models were generally the least truthful, because they imitate the falsehoods in their texts better (“inverse scaling”).

In code

def bins(conf, ok, n_bins=5):
    """Group by confidence (equal-width bins) and return (ECE, [(n, mean confidence, mean accuracy), ...])."""
    rows, ece = [], 0.0
    for b in range(n_bins):
        lo, hi = b / n_bins, (b + 1) / n_bins
        idx = [i for i, c in enumerate(conf) if lo <= c < hi or (b == n_bins - 1 and c == 1.0)]
        if not idx:
            continue
        c_mean = sum(conf[i] for i in idx) / len(idx)
        a_mean = sum(ok[i] for i in idx) / len(idx)
        ece += len(idx) / len(conf) * abs(a_mean - c_mean)
        rows.append((len(idx), round(c_mean, 2), round(a_mean, 2)))
    return round(ece, 3), rows


def threshold(conf, ok, t):
    """Answer only if confidence >= t. Returns (right, wrong, silent)."""
    answered = [o for c, o in zip(conf, ok) if c >= t]
    return sum(answered), len(answered) - sum(answered), len(conf) - len(answered)


conf = [0.97, 0.91, 0.83, 0.62, 0.55, 0.38, 0.31, 0.22, 0.12, 0.08]
ok   = [1,    1,    1,    1,    0,    1,    0,    0,    1,    0]
print(bins(conf, ok))
# (0.211, [(2, 0.1, 0.5), (3, 0.3, 0.33), (1, 0.55, 0.0), (1, 0.62, 1.0), (3, 0.9, 1.0)])
print(threshold(conf, ok, 0.0))   # (6, 4, 0)
print(threshold(conf, ok, 0.5))   # (4, 1, 5)

What comes next

Retrieval (lesson 31) attacks the root of the problem: the fact missing from the model’s memory. Evaluating a system on your own cases, not on a demo, is lesson 34. The risks that go beyond quality (privacy, prompt injection, accountability) are in lesson 35.