Layers and activations

A layer, in one formula

A layer with $m$ neurons and $n$ inputs stores a weight matrix $W\in\mathbb{R}^{m\times n}$ (one row per neuron) and a bias vector $\mathbf{b}\in\mathbb{R}^m$. Given the input $\mathbf{x}$, it computes

where $\phi$ is the activation, applied component by component. A forward pass chains this formula layer by layer. A dense network with sizes $n_0, n_1,\dots,n_L$ has $\sum_{l=1}^{L}(n_{l-1}+1)\,n_l$ parameters: for example, the 2-8-8-8-1 network of the playground has $3\cdot 8 + 9\cdot 8 + 9\cdot 8 + 9\cdot 1 = 177$.

The linear collapse

If $\phi$ is the identity, two layers give

which is a single layer with weights $W_2W_1$ and bias $W_2\mathbf{b}_1+\mathbf{b}_2$. With the numbers of the lesson:

By induction, $L$ linear layers equal one. With $\phi=\mathrm{ReLU}$ and input $(0,1)$, the hidden layer gives $(2,-1)$, ReLU turns it into $(2,0)$, and the output is $2\cdot2+1\cdot0=4$, not $3$: the equivalence breaks.

The activations

Their derivatives matter for training (lesson 8): $\sigma'(z)=\sigma(z)(1-\sigma(z))\le \tfrac14$, $\tanh'(z)=1-\tanh^2(z)\le 1$, and $\mathrm{ReLU}'(z)\in\{0,1\}$ (at $z=0$ it does not exist; by convention it is taken as 0). ReLU, the “rectified linear unit”, has smooth variants such as GELU (Hendrycks and Gimpel, 2016), which BERT uses instead of ReLU (Devlin et al., 2019).

Universal approximation

What it says (Cybenko, 1989, theorem 2). A function $\sigma$ is sigmoidal if $\sigma(t)\to1$ as $t\to+\infty$ and $\sigma(t)\to0$ as $t\to-\infty$. If $\sigma$ is continuous and sigmoidal, then for every continuous function $f$ on the unit cube $K=[0,1]^n$ (rescaling, on any cube) and every $\varepsilon>0$ there exist $N$, $\alpha_j$, $\mathbf{w}_j$ and $b_j$ such that

Who else, and under which conditions. The same year, Hornik, Stinchcombe, and White proved it for “squashing functions” (non-decreasing, with limits 0 and 1, not necessarily continuous), and for measurable functions in a weaker sense (approximation in probability); in a closing note they acknowledge Funahashi’s work (1989), which had reached a close result independently. The hypotheses do not coincide: Cybenko does not require $\sigma$ to be monotone, and Hornik et al. do not require it to be continuous. Hornik (1991) showed that a bounded, nonconstant activation is enough ($L^p$ approximation; if it is also continuous, uniform approximation on compact sets). Leshno, Lin, Pinkus, and Schocken (1993) settled the question: for a locally bounded, piecewise continuous activation, a network with one hidden layer approximates every continuous function, uniformly on every compact set, if and only if the activation is not a polynomial. ReLU meets that condition. The theorem also needs the bias (the threshold): without it, it fails.

What it does not say. It approximates a known function on a bounded set; it says nothing outside it, nor about learning from examples. It does not bound $N$ or say how to find the weights. Cybenko admits this in his closing discussion: his paper is limited to existence, and he suspects that the vast majority of approximation problems will need an astronomical number of terms, because of the curse of dimensionality. Hornik et al. add that a failure in practice may be due to inadequate learning, too few neurons, or a random relation between input and output.

The construction in the lesson. It is not Cybenko’s proof, which uses the Hahn–Banach theorem and builds nothing. For a function $f$ of one variable, with $n$ equally spaced knots $t_k$ and slopes $s_k=\frac{f(t_k+h)-f(t_k)}{h}$, $h=(b-a)/n$,

is a one-hidden-layer network with $n$ ReLU neurons that equals $f$ at the knots (piecewise linear interpolation). If $|f''|\le M$, the maximum error is at most $\dfrac{(b-a)^2M}{8n^2}$: doubling the neurons divides the bound by four. For the scene’s curve, $M\le 3.36$; with $n=24$ the bound is 0.026 and the measured error is 0.025. The ratio of measured errors is not always four: 2.5 from 3 to 6 neurons, 3.0 from 6 to 12, 4.1 from 12 to 24, and 3.8 from 24 to 48. Nor does it drop at every step (from 1 to 2, 4 to 5, and 7 to 8 neurons it rises a little, because equally spaced knots fall in other places). The turn’s goal, 0.05, is reached with 17 neurons (0.047; with 16, 0.057). Here the weights are computed, not trained.

Depth: the sawtooth

The tent function $g(x)=2\,\mathrm{ReLU}(x)-4\,\mathrm{ReLU}(x-\tfrac12)$ on $[0,1]$ uses two neurons: it rises from 0 to 1 on $[0,\tfrac12]$ and falls from 1 to 0 on $[\tfrac12,1]$. The composition $g^{(k)}=g\circ\cdots\circ g$ ($k$ layers, $2k$ neurons) has $2^k$ straight pieces (checked numerically up to $k=6$). A network with one hidden layer of $m$ ReLU neurons has at most $m+1$ pieces in one dimension (each neuron adds one corner), so to match it, it needs at least $2^k-1$ neurons. Telgarsky (2016) uses this same function and proves something stronger, for approximating and not just matching: for every $k$ there is a ReLU network with $2k^3+8$ layers and $3k^3+12$ neurons such that every ReLU network with at most $k$ layers and at most $2^k$ neurons stays at $L^1$ distance at least $1/64$ on $[0,1]^d$. It is a result about specific functions, not a law: it does not say that every real task benefits equally from depth.

In code: the forward pass and the collapse

import numpy as np

W1 = np.array([[1, 2],
               [3, -1]])        # hidden layer: one row per neuron
W2 = np.array([[2, 1]])         # output

def net(x, activation):
    h = activation(W1 @ x)      # forward pass: hidden layer...
    return (W2 @ h)[0]          # ...and output

linear = lambda z: z
relu = lambda z: np.maximum(0, z)

print("W2 @ W1 =", W2 @ W1)     # the equivalent single neuron
for x in ([2, 1], [1, 0], [0, 1]):
    x = np.array(x)
    print(x, "no activation:", net(x, linear), "| one neuron:", (W2 @ W1 @ x)[0], "| with ReLU:", net(x, relu))

Expected output:

W2 @ W1 = [[5 3]]
[2 1] no activation: 13 | one neuron: 13 | with ReLU: 13
[1 0] no activation: 5 | one neuron: 5 | with ReLU: 5
[0 1] no activation: 3 | one neuron: 3 | with ReLU: 4

In code: a network that learns the circles

It is the same idea as the playground (a 2-6-1 network, ReLU and a sigmoid output), with plain gradient descent. The lines with dZ are backpropagation, explained in lesson 8.

import numpy as np
rng = np.random.default_rng(0)

# Data: a central group (class 0) surrounded by a ring (class 1)
n = 90
a0, r0 = rng.uniform(0, 2 * np.pi, n), np.sqrt(rng.uniform(0, 1, n)) * 1.15
a1, r1 = rng.uniform(0, 2 * np.pi, n), rng.uniform(1.95, 2.8, n)
X = np.vstack([np.c_[r0 * np.cos(a0), r0 * np.sin(a0)], np.c_[r1 * np.cos(a1), r1 * np.sin(a1)]])
X = (X + 0.12 * rng.normal(size=X.shape)) / 3          # coordinates in [-1, 1]
y = np.r_[np.zeros(n), np.ones(n)]

# Network 2 → 6 (ReLU) → 1 (sigmoid)
W1 = rng.normal(0, np.sqrt(2 / 2), (2, 6)); b1 = rng.uniform(-0.5, 0.5, 6)
W2 = rng.normal(0, np.sqrt(1 / 6), (6, 1)); b2 = np.zeros(1)
sig = lambda z: 1 / (1 + np.exp(-z))

for step in range(1500):
    Z1 = X @ W1 + b1; H = np.maximum(0, Z1); p = sig(H @ W2 + b2).ravel()   # forward pass
    dZ2 = (p - y)[:, None] / len(y)                                          # error at the output
    dW2, db2 = H.T @ dZ2, dZ2.sum(0)
    dZ1 = (dZ2 @ W2.T) * (Z1 > 0)                                            # the blame flows back to the hidden layer
    dW1, db1 = X.T @ dZ1, dZ1.sum(0)
    for P, G in ((W1, dW1), (b1, db1), (W2, dW2), (b2, db2)):
        P -= 0.5 * G                                                          # gradient descent
print("accuracy:", ((p > 0.5) == y).mean())

Expected output: accuracy: 1.0. With 6 neurons it gave 1.0 on all ten seeds tried (0 to 9); with 3 neurons, between 0.81 and 1.0; with 2, between 0.82 and 0.89 (NumPy’s random numbers may change between versions). These are hits on the same points used for training: being right on new data is a different question (lesson 10). The lesson’s playground uses the Adam optimizer instead of plain descent (lesson 9).

What comes next

Computing the gradient of every weight in a network like this is backpropagation (lesson 8). How to train well —batches, optimizers— is lesson 9, and lesson 10 shows that being right on the training points, as here, does not guarantee being right on new data.