Evaluating AI systems

The margin of a score

With $k$ right answers in $n$ cases, the measured accuracy is $\hat p = k/n$. Its approximate 95% confidence interval is

With 78 of 100: $0.78 \pm 1.96\sqrt{0.78 \cdot 0.22/100} = 0.78 \pm 0.081$, that is, from 70% to 86% (what the scene uses). With 400 cases, $\pm 0.041$. The margin goes with $1/\sqrt n$: quadrupling the cases halves it, and in the worst case ($\hat p = 0.5$) it is almost exactly $1/\sqrt n$: 10 points with 100 cases, 5 with 400, 2.5 with 1,600.

The correct reading: if you repeated the evaluation many times with fresh cases, about 95% of the intervals built this way would contain the true accuracy. With few cases, or scores near 0% or 100%, this formula falls short; Brown, Cai, and DasGupta (2001) recommend the Wilson interval, which for 78 of 100 gives 68.9% to 85.0%. Bowyer, Aitchison, and Ivanova (2025) show that with fewer than a few hundred cases, margins based on the normal approximation tend to come out too small.

Comparing two systems on the same cases

For each case $i$, let $d_i = a_i - b_i \in \{-1, 0, +1\}$ (A's hit minus B's hit). If only A is right on $n_{10}$ cases and only B on $n_{01}$, the measured difference is $\bar d = (n_{10}-n_{01})/n$ and its standard error is

In the lesson's simulation, on each case both are right with probability 0.66, only A 0.12, only B 0.08, and neither 0.14. The variance of $d_i$ is $0.12 + 0.08 - 0.04^2 = 0.198$; if each system were tested on different cases, their variances would add up, $0.78\cdot0.22 + 0.74\cdot0.26 = 0.364$: almost double. That is why, with the same cases, you need almost half as many.

The exact probability that A comes out ahead (from the multinomial distribution) is 78.5% with 100 cases, 88.4% with 200, 95.9% with 400, and 99.75% with 1,000; the scene's 100 comparisons give 78, 88, 96, and 100. With different cases, 400 cases give about 90%, and it takes about 650 to reach 95%; with the same cases, about 360 are enough. A winning is not the same as proving the difference: for the interval of $\bar d$ to exclude zero 80% of the time takes $n \approx (1.96 + 0.84)^2 \cdot 0.198/0.04^2 \approx 970$ cases.

Miller (2024) sums up the craft: report standard errors, compare in pairs (case by case), cluster cases that are not independent (for example, several questions about the same document), and plan the sample size before evaluating.

Contamination: the example's arithmetic

If a model always gets right what it saw, solves a fraction $a$ of the rest, and saw a fraction $c$ of the exam, its score is

With $a = 0.70$: $c = 0.2$ gives $0.2 + 0.8 \cdot 0.7 = 0.76$, the same as B, and $c = 0.4$ gives $0.82$. It is a simplified model: real memory is partial. Contamination is searched for with text matches between the exam and the corpus (the GPT-4 report used three 50-character substrings per question; that is how it found that part of BIG-bench had slipped into training by accident, and it estimated that a quarter of HumanEval was in its data), with equivalent fresh problems, or by measuring how much probability the model assigns to the exam's questions. GSM1k (Zhang et al., 2024) is 1,205 problems written by people in the style of GSM8k: some model families dropped by up to 8 points, with bigger drops the more likely the model was to generate GSM8k examples; frontier models barely dropped.

The automatic judge

Zheng et al. (2023) compared GPT-4 as a judge with human experts on MT-bench: not counting ties, they agreed 85% of the time, more than the experts with each other (81%). They also measured its biases. Order: with two nearly identical answers, the verdict held when the order was swapped 65.0% of the time with GPT-4, 46.2% with GPT-3.5, and 23.8% with Claude-v1. Length: when 23 answers were padded with a repeated list that added no information, Claude-v1 and GPT-3.5 preferred the padded version in 21 of 23 (91.3%), and GPT-4 in 2 (8.7%). Their defense against order bias: grade twice, swapping the order, and declare an answer the winner only if it wins both times.

To measure agreement with people without being fooled by chance, Cohen's $\kappa$ is used: $\kappa = (p_o - p_e)/(1 - p_e)$, where $p_o$ is the observed agreement and $p_e$ the agreement expected by chance. Human evaluation is also done at scale: Chatbot Arena gathered over 240,000 votes from people comparing answers blind (Chiang et al., 2024), but on questions that are not yours.

Scores expire, and exams have flaws too

Chen, Zaharia, and Zou (2023) measured the same service in March and in June 2023 with 500 prime and 500 composite numbers: GPT-4 went from 84.0% to 51.1%, because the June version almost always answered “composite” (99.7%) and stopped following the step-by-step reasoning it was asked for. It is a change in behavior more than in ability, but for anyone who depends on the service the effect is the same, and you only notice by measuring again.

Exams contain errors: Gema et al. (2024) estimate that 6.49% of MMLU questions have one (57% of those reviewed in virology). And they run out: Phan et al. (2025) built a new 2,500-question exam because the best models already scored above 90% on MMLU.

In code

import numpy as np

def margin(k, n, z=1.96):
    """Half the 95% confidence interval for k right answers out of n cases."""
    p = k / n
    return z * np.sqrt(p * (1 - p) / n)

for n in (100, 400, 1600):
    print(n, round(100 * margin(0.78 * n, n), 1))
# 100 8.1
# 400 4.1
# 1600 2.0

# Two systems on the same cases: both right 0.66; only A, 0.12; only B, 0.08; neither, 0.14
rng = np.random.default_rng(0)
def a_ahead(n, reps=20000):
    c = rng.multinomial(n, [0.66, 0.12, 0.08, 0.14], size=reps)   # both, only A, only B, neither
    return round(float(np.mean(c[:, 1] > c[:, 2])), 2)

for n in (100, 200, 400, 1000):
    print(n, a_ahead(n))
# 100 0.78
# 200 0.88
# 400 0.96
# 1000 1.0

What comes next

Measured quality is one axis; risk is another: privacy, manipulation through instructions, bias, and regulation. Lesson 35 walks through them, and lesson 36 uses this evaluation inside a pilot that scales.