A model and its error

The line, piece by piece

A straight line is the graph of $\hat y = wx + b$ (read “y-hat”: the prediction). The weight $w$ is the slope: if the input grows by $\Delta x$, the prediction grows by $w\,\Delta x$, so $w = \Delta \hat y / \Delta x$, rise over run. The bias $b$ is the intercept: $\hat y(0) = b$. In the lesson, $w = 5$ and $b = 40$: with 7 hours, $\hat y = 5\cdot 7 + 40 = 75$.

The model and the loss

With several features (lesson 2) the input is a vector $\mathbf{x}$ and the prediction is $\hat y = \mathbf{w}\cdot\mathbf{x} + b$: one weight per feature and a single bias. The ten students in the lesson are invented.

The error of a point is $e_i = y_i - \hat y_i$. With $n$ points, the two losses in the lesson are

The MSE is measured in squared points. Its root, $\sqrt{L}$, goes back to points: with the best line in the lesson, $\sqrt{42.5}\approx 6.5$ points of typical error. The example line ($w = 5$, $b = 40$) has $L = 45.775$; the flat line at the average, $L = 244.04$.

Why the square

• It is smooth. $L(w,b)$ is a paraboloid: it has a single bottom and a well-defined slope everywhere, which will let us slide down it (lesson 4). The absolute value has a kink at zero.
• It has a formula. Setting the partial derivatives of $L$ to zero gives a linear system, the normal equations.
• It has a probabilistic justification. If $y_i = wx_i+b+\varepsilon_i$ with normal noise $\varepsilon_i\sim\mathcal N(0,\sigma^2)$, the likelihood is highest exactly where the MSE is lowest (Goodfellow et al., 2016, sec. 5.5.1, which calls it a criterion introduced “more or less arbitrarily” before justifying it this way). Gauss set this out in 1809 (Theoria motus, arts. 175–179): if the errors follow the normal curve, the most probable system of values is the one with the smallest sum of squares. With Laplace noise, which has heavier tails, the matching criterion is the absolute error.
• It is sensitive to outliers. With a single parameter (predicting a constant), the MSE is minimized by the mean and the absolute error by the median. In the lesson’s data, the mean is 64.4 and the median 59.5 (with ten data points, any value between 57 and 62 minimizes the absolute error; the midpoint is taken).

A little history

Legendre presented least squares in 1805, in an appendix, as the principle that in his judgment is the “most general, most exact and of easiest application”, and that sets up among the errors “a sort of equilibrium” that keeps the extremes from prevailing. Gauss wrote in 1809 that he had used it since 1795 (Theoria motus, art. 186) and that, among all even powers of the error, the square was “the most simple”; the priority was disputed (Plackett, 1972; Stigler, 1981).

In the same article Gauss reports that Laplace used the absolute error, proposed earlier by Boscovich. He notes that with it the line generally passes through two of the points, and that any other value could have been observed for another point without changing the solution, provided it stays on the same side: it is what you saw with the outlier, whose absolute line did not move when its grade was lowered. Between the two extremes there are hybrid losses, such as Huber’s (1964): quadratic for small errors and linear for large ones.

The best line

With $\bar x = 4.55$ and $\bar y = 64.4$ we get $w^*\approx 5.26$ and $b^*\approx 40.45$, with minimum loss $L_{\min}\approx 42.5$. The best line always passes through the mean point $(\bar x,\bar y)$. In matrix form these are the normal equations, $\mathbf{w}^*=(X^\top X)^{-1}X^\top\mathbf{y}$ (in $X$, a column of ones takes care of the bias).

If we keep $b=b^*$ and move only $w$, the loss is exactly a parabola:

It is the valley in the last scene. The name “regression” comes from Galton. In his 1886 study of stature (930 adult children from 205 families) he found that the children’s height deviation is, on average, two thirds of their parents’: the children of very tall parents tend to be less tall, and those of very short parents, taller. He called it “regression towards mediocrity”, that is, towards the average.

In code

# ten invented students for the lesson
hours = [0.5, 1.5, 2, 3, 4, 5, 6, 6.5, 8, 9]
grade = [43, 49, 52, 57, 62, 72, 73, 56, 87, 93]
n = len(hours)
mx, my = sum(hours) / n, sum(grade) / n

# least squares: w = cov(x, y) / var(x),  b = mean(y) - w * mean(x)
w = sum((x - mx) * (y - my) for x, y in zip(hours, grade)) / sum((x - mx) ** 2 for x in hours)
b = my - w * mx

def loss(w, b):  # mean squared error (MSE)
    return sum((y - (w * x + b)) ** 2 for x, y in zip(hours, grade)) / n

print(round(w, 2), round(b, 2))   # 5.26 40.45   (best weight and bias)
print(round(loss(w, b), 1))       # 42.5         (minimum loss)
print(round(loss(0, my), 1))      # 244.0        (flat line: always predict the mean)
print(round(loss(5, 40), 1))      # 45.8         (the example line: weight 5, bias 40)

What comes next

The closed formula works for a straight line. In a network with billions of weights there is none, and the bottom is found step by step: that is gradient descent (lesson 4). And lowering the loss on the training data is not enough: it has to be measured on data the model has not seen (lesson 10).