The artificial neuron

The neuron as a formula

A neuron with inputs $x_1,\dots,x_n$, weights $w_1,\dots,w_n$ and bias $b$ computes

where $f$ is the activation function. With the lesson’s numbers, $\mathbf{x}=(0.8,\,0.2)$, $\mathbf{w}=(2,\,-1.5)$ and $b=-0.5$:

With the step, $y=1$. With the sigmoid, $y=\sigma(0.8)\approx 0.69$. If $n=1$ and $f$ is the identity, we get $y=wx+b$: the line from lesson 3 is the simplest case of a neuron.

The decision boundary

The neuron changes its answer where $z=0$, that is, $w_1x_1+w_2x_2+b=0$. It is a line whose normal vector is $\mathbf{w}$. Its distance from the origin is $|b|/\lVert\mathbf{w}\rVert$, and the signed distance from a point to it is $z/\lVert\mathbf{w}\rVert$. In the example, $\lVert\mathbf{w}\rVert=2.5$: the line passes $0.2$ from the origin and the example email sits $0.8/2.5=0.32$ away.

Changing $b$ shifts the line; changing $\mathbf{w}$ rotates it (if $w_2=0$ it is vertical: only $x_1$ matters). Multiplying $\mathbf{w}$ and $b$ by $k>0$ does not move the line, but with a sigmoid it changes how sharp the transition is, because the output becomes $\sigma(kz)$. With $n$ inputs, the boundary is a hyperplane.

Two activations

The step is $H(z)=1$ if $z\ge 0$ and $0$ otherwise. The sigmoid is

with maximum slope $1/4$ at $z=0$. The derivative of the step is zero except at $z=0$, where it does not exist: it gives gradient descent no signal at all. The sigmoid’s derivative exists everywhere. A sigmoid neuron trained to minimize cross-entropy is logistic regression. Other activations are also used today, such as ReLU (lesson 7).

The perceptron rule

With $t,y\in\{0,1\}$ (the correct answer and the neuron’s answer):

It changes only when $t\neq y$, and then $t-y=\pm1$. The same example improves: $\Delta z=\eta\,(t-y)\,(\lVert\mathbf{x}\rVert^2+1)$ has the right sign. It is a stochastic subgradient descent step on the loss $\max(0,-\tilde t\,z)$ (the “perceptron criterion”), with $\tilde t=2t-1$: the idea of lesson 4, applied to a different loss because the step’s own loss (counting errors) is flat almost everywhere.

The convergence theorem says: if there is a unit vector $\mathbf{u}$ and a margin $\gamma>0$ with $\tilde t_i\,(\mathbf{u}\cdot\mathbf{x}_i)\ge\gamma$ for every example (with the bias as one more input, equal to 1) and $\lVert\mathbf{x}_i\rVert\le R$, the perceptron, starting from zero weights, makes at most $(R/\gamma)^2$ mistakes: the bound comes straight out of Novikoff’s proof (1962). With the lesson’s 16 emails, $R\approx 1.63$ (the largest $\lVert(x_1,x_2,1)\rVert$) and $\gamma\approx 0.12$, so the bound is about 190 mistakes; in 300 runs with random order, from zero, it never took more than 31. If the data cannot be separated by a line, it does not converge: the weights never stop moving.

In code

import math

def neuron(x, w, b, activation="step"):
    z = sum(wi * xi for wi, xi in zip(w, x)) + b
    if activation == "step":
        return 1 if z >= 0 else 0
    return 1 / (1 + math.exp(-z))

x, w, b = [0.8, 0.2], [2.0, -1.5], -0.5
print(round(sum(wi * xi for wi, xi in zip(w, x)) + b, 2))   # z
print(neuron(x, w, b))                                       # step
print(round(neuron(x, w, b, "sigmoid"), 3))                  # sigmoid
# 0.8
# 1
# 0.69

And the perceptron rule learning the OR function:

data = [([0, 0], 0), ([0, 1], 1), ([1, 0], 1), ([1, 1], 1)]
w, b, eta = [0.0, 0.0], 0.0, 1.0

def output(x):
    return 1 if w[0] * x[0] + w[1] * x[1] + b >= 0 else 0

for epoch in range(10):
    errors = 0
    for x, t in data:
        y = output(x)
        if y != t:
            w[0] += eta * (t - y) * x[0]
            w[1] += eta * (t - y) * x[1]
            b += eta * (t - y)
            errors += 1
    print(epoch, errors, w, b)
    if errors == 0:
        break
# 0 2 [0.0, 1.0] 0.0
# 1 2 [1.0, 1.0] 0.0
# 2 1 [1.0, 1.0] -1.0
# 3 0 [1.0, 1.0] -1.0

Where it comes from, carefully

McCulloch and Pitts (1943) assumed all-or-none neurons that fire when a fixed number of excitatory synapses is active, with absolute inhibition (a single inhibitory synapse prevents firing) and a network that does not change over time. They showed that such networks can compute functions of propositional logic. They had no weights to be learned.

Rosenblatt (1958) described the perceptron as a system of threshold units: a retina of sensors (S), association units (A) wired at random, and response units (R). The paper describes learning as reinforcement of the A units (the experimenter “forces” the response, or rewards and punishes); it does not contain the rule above. Correcting only when there is an error appears in his laboratory’s 1960 reports, and the single-neuron form is the one taught since.

The Mark I Perceptron (1960 manual): 400 photoresistors in a 20 × 20 grid, 512 A units wired at random to the sensors (generated by a program on an IBM 704 and set on a plugboard), and 8 R units. The values of the A units, the weights, are potentiometers moved by direct-current motors.

The July 1958 demonstration ran on an IBM 704 computer: after 50 trials it told cards marked on the left from cards marked on the right. The Navy’s line about a machine “conscious of its existence” comes from the UPI dispatch that The New York Times published on July 8.

The brain numbers. Azevedo et al. (2009) counted, in adult male brains, $86.1\pm 8.1$ billion neurons on average. Beniaguev et al. (2021) needed a temporally convolutional network with five to eight layers to imitate a detailed biophysical model of a layer 5 cortical pyramidal neuron; when they removed the NMDA receptors from the model, a network with a single hidden layer was enough. It is a property of that simulated model, not a measure of every neuron.

What comes next

A single neuron can only separate with a line; lesson 6 shows a case it cannot handle (XOR). Lesson 7 stacks layers with nonlinear activations, and lesson 8 computes the gradient of every weight in the network (backpropagation).