Generalizing and evaluating

Bias, variance, and noise

For a model $\hat f$ trained on a random set, the expected error at a new point decomposes into

This is the standard form for squared error (Hastie, Tibshirani, and Friedman, eq. 7.9). Underfitting is high bias (a model that is too rigid); overfitting is high variance (the model changes a lot with each sample). In the lesson the noise is $\sigma = 0.2$: on average, no model can go below that error on new data (on one particular 80-point sample it can land a little above or below), and the best degree (4) reaches $\approx 0.22$ on the scene's validation set.

This was checked by simulating 4,000 training sets with the same data recipe (verify-data.ts): the sum $\sigma^2+\text{bias}^2+\text{variance}$ matches the simulated error at new points. With degree 1: bias² ≈ 0.066 and variance ≈ 0.007 (the error is bias). With degree 4: 0.001 and 0.017. With degree 10: bias² ≈ 0 and variance ≈ 1.6, mostly toward the ends of the interval (the error is variance). With degree 10 and $\lambda = 0.03$: 0.005 and 0.018; regularization trades variance for a little bias.

Ridge regularization

Polynomial fitting with a penalty $\lambda$ on the weights minimizes $\lVert y - Xw\rVert^2 + \lambda \lVert w\rVert^2$, with closed-form solution

With $\lambda = 0$ you recover the free fit; as $\lambda$ grows the weights shrink and the curve smooths out (in the scene, the norm of the degree-10 weights drops from ≈ 327 at $\lambda = 0$ to ≈ 0.19 at $\lambda = 10$). Two caveats. The constant term is not penalized (as in the scene and in scikit-learn; Hastie et al., §3.4.1), and inputs should be normalized first (the scene maps $x\in[0,1]$ to $[-1,1]$). And with $\lambda$ too large the model underfits again: at degree 10, the scene's validation error falls until $\lambda \approx 0.02$ (≈ 0.24) and then rises (≈ 0.44 at $\lambda = 10$, worse than the line).

Early stopping in gradient descent acts similarly to regularization: stopping earlier limits how large the weights can grow. For a linear model with squared error and plain gradient descent it is equivalent, under a quadratic approximation, to $L^2$ regularization (Goodfellow, Bengio, and Courville, §7.8). Other defenses: more data (real or augmented, §7.4) and, in neural networks, dropout: randomly switching off units during training (Srivastava et al., 2014).

Classification metrics

With true positives $TP$, false positives $FP$, false negatives $FN$, and true negatives $TN$:

If nobody is flagged ($TP+FP=0$) precision is $0/0$: by convention (the scene’s) it is taken as 1, so a detector that always says “healthy” has perfect precision and recall 0, which is why you have to look at both (Goodfellow et al., §11.1). $F_1$ is the harmonic mean of precision and recall.

The threshold and the costs

If a false negative costs $c_{FN}$ and a false positive costs $c_{FP}$ (hits cost nothing), the lowest-expected-cost decision is to flag “positive” when

With equal costs the probability cutoff is 0.5; if missing a case costs 10 times more, it drops to $1/11\approx 0.09$. In the scene (two bell curves and 10% sick) that gives the optimal score thresholds 2.23 (equal costs), 1.46 (missing a case costs 10), and 3.00 (a false alarm costs 10).

The base rate (Bayes’ theorem)

With prevalence $p$, sensitivity $s$ (= recall), and specificity $c$, the probability of being sick given a positive result is

With $p=0.01$ and $s=c=0.95$: $\dfrac{0.0095}{0.0095+0.0495}\approx 0.16$, the “one in six” of the lesson. With $s=c=0.95$ there are more false alarms than hits as long as the prevalence is below 5%.

Presenting the calculation in frequencies (“out of 2,000 people, 20 sick, 19 detected…”) instead of probabilities helps reasoning: Gigerenzer and Hoffrage (1995) show that Bayesian algorithms are computationally simpler in that format and that, with it, people without statistical training followed Bayesian algorithms in up to 50% of their inferences. It is the format the lesson uses.

In code

import numpy as np
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import PolynomialFeatures
from sklearn.linear_model import Ridge

f = lambda x: 0.6 * np.sin(2 * np.pi * x) + 0.2          # the true signal
rng = np.random.default_rng(0)
x_tr = ((np.arange(12) + 0.5) / 12)[:, None]              # 12 measurements spread over [0, 1]
y_tr = f(x_tr[:, 0]) + 0.2 * rng.normal(size=12)
x_va = rng.uniform(0, 1, 80)[:, None]                     # 80 new measurements
y_va = f(x_va[:, 0]) + 0.2 * rng.normal(size=80)

for degree in (1, 4, 10):
    for lam in (1e-9, 3e-2):                              # 1e-9 stands in for λ = 0
        m = make_pipeline(PolynomialFeatures(degree, include_bias=False), Ridge(alpha=lam)).fit(2 * x_tr - 1, y_tr)
        rmse = lambda a, b: float(np.sqrt(np.mean((m.predict(2 * a - 1) - b) ** 2)))
        print(degree, lam, round(rmse(x_tr, y_tr), 3), round(rmse(x_va, y_va), 3))   # training and validation

# With that seed: degree 1 → 0.302 and 0.339 (underfitting); degree 4 → 0.093 and 0.221 (best validation);
# degree 10 → 0.006 and 0.251 with λ ≈ 0 (overfitting) and 0.103 and 0.237 with λ = 0.03.
# These are new data from the same recipe as the scene: they do not repeat its exact numbers.

def ppv(prev, sens, spec):          # positive predictive value
    tp, fp = prev * sens, (1 - prev) * (1 - spec)
    return tp / (tp + fp)

print(round(ppv(0.01, 0.95, 0.95), 3))   # 0.161

What comes next

For images, a fully connected network does not scale. Lesson 11 shows the idea of convolutions. Lesson 34 returns to evaluation for generative AI systems, where measuring well is harder.