One neuron, by hand.
Defaults are the class's one worked example: x = (2, 1), w = (0.5, −1.0), b = 0.25 gives z = 0.25 and a = σ(z) = 0.562177.
Everything you trained in your CNNs was built out of one small part, repeated millions of times. Tonight we take one of them out, put it on the bench, and work it by hand: what it computes, what its knobs mean geometrically, why it needs a smooth activation to learn, how one gradient step improves it, and the exact moment it fails. You'll derive every formula yourself and check every number against the page.
The figure above is the whole class in one machine. Two input carriages slide on their rails (x₁, x₂). Each signal goes through a weight knob (w₁, w₂) that scales it, possibly flipping its sign. The summing drum adds the weighted signals and the push of the bias spring (b) into one number, z. The activation cam turns z into the output a, which the dial reads between 0 and 1. Hover any part to see the math it implements. Then scroll: each section below takes one part of the bench and takes it apart.
A neuron weighs evidence
In 1943 Warren McCulloch and Walter Pitts made a bold simplification. A biological neuron either fires or it doesn't, so treat it as a logic device: it adds up the signals arriving on its inputs, and if the total passes a threshold it fires. Their paper opens with exactly that idea:
Strip away the biology and what's left is arithmetic. Each input xⱼ is multiplied by a weight wⱼ, a number saying how much that input matters and in which direction. The products are summed and compared with a threshold:
Michael Nielsen's textbook gives the friendliest version: deciding whether to go to a cheese festival. Three yes/no facts are the inputs: is the weather good, will your partner come, is it near transit? You care most about the weather, so it gets weight 6 and the others get 2. With a threshold of 5, the weather alone decides. Lower the threshold to 3 and you'll also go when your partner comes and it's near transit. Same evidence, different decision, just by moving one number.
Two things to take away. First, the weights are the knowledge: change them and the same machine makes different decisions. Second, McCulloch and Pitts set the weights by hand; nothing in their neuron learns. Making the weights learn, and making them learnable by calculus, is the rest of tonight. Your CNN's conv filter is this same unit, slid across every position of the image with its weights shared.
Weights (6, 2, 2) and thresholds 5 and 3 are Nielsen's example. The table below lists every input pattern, with the ones that fire marked.
A neuron draws a line
First, a notational move that pays off forever. Move the threshold to the other side of the inequality and call its negative the bias, b ≡ −threshold. Then the neuron computes one number,
and says "yes" when z > 0. The bias is how easily the neuron fires: a large positive b means yes almost regardless of the evidence.
Now look at it geometrically. The points where z = 0 form a straight line in the input plane (a hyperplane in more dimensions). Every point on one side gives z > 0, every point on the other z < 0. Two facts are worth deriving yourself. The weight vector w is perpendicular to the line: if two points p, q are both on the line, then w·(p − q) = 0. And it points toward the "yes" side. The line's distance from the origin is
For AND, the weights w = (1, 1) with b = −1.5 put the line 1.06066 from the origin, with only (1, 1) on the yes side. The same weights with b = −0.5 give OR.
Here's the surprise to test in the figure. Press × 10. Multiplying w and b by 10 doesn't move the line at all, because 10z = 0 exactly where z = 0. What changes is how sharply the output switches. The shading is the sigmoid output we'll meet in the next section. At (1, 1) it goes from 0.622459 to 0.993307. The direction of w chooses the line; its length chooses how confident the neuron is. Keep that in mind: when training pushes the weights to grow without bound, this is what it's doing.
Switch to the blobs set: 200 points from two Gaussian clouds, the same data as your lab. You can probably drag a line that gets almost all of them right. That's a neuron's whole repertoire: one straight cut through input space. Your CNN's last layer is exactly this, a set of such cuts through the feature space the conv layers built.
Why the step can't learn
McCulloch and Pitts' neuron uses a step: output 1 if z > 0, else 0. That's a perfect logic gate and a terrible student. To learn, we want to nudge a weight and see the output improve a little. But the step's slope is zero everywhere it exists (and undefined at 0). A small nudge changes nothing, until suddenly an answer flips. Gradient-based learning asks "which way is downhill?" and the step answers "flat", everywhere.
The fix is a smoothed step, the sigmoid:
Derive its slope yourself. Write σ = (1 + e^(−z))^(−1) and differentiate:
That identity is why the sigmoid is so convenient: once you've computed the output a, its slope is just a(1 − a), with no extra exponentials. The slope peaks at 0.25 at z = 0. At our worked example, z = 0.25, it's 0.246134.
Drag the marker out to z = ±5 and watch the slope collapse toward zero. That's saturation: a neuron that is very sure of itself barely responds to nudges. It's not a bug of the sigmoid; it's the price of being smooth and bounded. In the next section you'll see how the right choice of loss cancels it out exactly.
And ReLU, which your CNNs used? Its slope is 1 for z > 0 and 0 for z < 0. It's not bounded, but it doesn't saturate on the positive side, which is why it took over hidden layers. For one output neuron that reads as a probability, the sigmoid is the right tool, so that's what we'll train tonight.
The solid curve is the activation and the dashed curve is the sigmoid's slope σ′ = σ(1 − σ). ReLU is clipped at 1.2 on this axis.
Measure the error, follow the chain rule, take one step
To learn, the neuron needs a number that says how wrong it is: the loss. Read the output a as the neuron's probability that the label is y = 1. The probability it gives the true label is a^y (1 − a)^(1−y), and we want that to be large. Taking the negative log turns "maximize the probability" into "minimize a sum", and gives binary cross-entropy:
Our worked example has label y = 1 and output 0.562177, so L = −ln a = 0.575939.
To improve, we need ∂L/∂w: how the loss changes when we turn each knob. The loss depends on a, a on z, and z on w, so the chain rule splits it into links:
Look at what happens: the a(1 − a) factors cancel, leaving the cleanest result in the subject,
The gradient is error times input. With our numbers, a − y = −0.437823, so ∂L/∂w = (−0.875647, −0.437823). A finite-difference check agrees to about 10⁻¹⁰.
The learning rule is then one line: step each knob against its gradient, w ← w − η ∂L/∂w. Near the current weights, the loss changes by about ∇L·Δθ, so this step changes it by about −η‖∇L‖² ≤ 0. With η = 0.5, our neuron goes from a = 0.562 to 0.826851, and the loss drops to 0.190131. How big η can safely be, and what the whole loss landscape looks like, is Class 2.
Why not plain squared error, ½(a − y)²? Its gradient is (a − y)·a(1 − a): the saturating slope does not cancel. Put a neuron at z = −5 when the answer is 1 (Fig 6). Squared error's gradient is −0.00660356, cross-entropy's is −0.993307: 150.42× larger. Cross-entropy shouts when the neuron is sure and wrong; squared error whispers.
Build a gate, then let it learn one
McCulloch and Pitts' big claim was that neurons compute logic. Check it with your own hands. AND says yes only for (1, 1). OR says yes unless both inputs are 0. NAND, Nielsen's favourite, is "not both": with weights (−2, −2) and bias 3, the input (1, 1) gives −2 − 2 + 3 = −1, a no, and every other input gives a yes. NAND is universal: any digital circuit can be built from NAND gates alone. So networks of these neurons can, in principle, compute anything a computer can.
Pick a gate in the figure and set w₁, w₂, b until all four rows of the truth table agree. A useful trick: decide first which corner must be alone on its side, then aim the line between that corner and the other three. Hand-designing works for four points, but it doesn't scale to the millions of weights in your CNN.
So press Train it. The page starts from w = b = 0, where the output is 0.5 for every input and the loss is ln 2 = 0.693147. Then it applies exactly the rule from the last section, w ← w − η · mean((a − y)x), to all four rows at once, 2000 times with η = 1. For AND it lands on w = (8.8215, 8.8215), b = −13.4026, with loss 0.00868 and all four rows right. The loss falls on every single iteration.
Notice two things about the learned weights. They're symmetric, because nothing in AND distinguishes x₁ from x₂, and gradient descent from a symmetric start can't break that. And they're large. Remember the × 10 button: the line was right long before training ended, and the rest of training just kept stretching w to push the outputs toward 0 and 1. On separable data, cross-entropy never stops rewarding confidence.
The one thing a single neuron can't do
XOR, "exactly one of the two", wants (0, 1) and (1, 0) on the yes side and (0, 0) and (1, 1) on the no side: the two diagonals of a square. Try it in the figure. You'll get three of four right easily, and never the fourth. A grid search over 61³ lines confirms the best any line can do is 0.75.
The proof takes four lines. Write what XOR demands of a step neuron:
- (0,0) → 0: b ≤ 0
- (1,0) → 1: w₁ + b > 0 and (0,1) → 1: w₂ + b > 0
- (1,1) → 0: w₁ + w₂ + b ≤ 0
- Add the two in line 2: w₁ + w₂ + 2b > 0, so w₁ + w₂ + b > −b ≥ 0. That contradicts line 3. ∎
Training can't escape a proof. Press Train it, and gradient descent on cross-entropy, starting from whatever line you left, slides to the least-bad compromise: w = 0, b = 0, output 0.5 for every input. At that point the XOR gradient is exactly zero, and since this loss is convex in (w, b), it's the global minimum. The loss settles at ln 2 = 0.693147, the loss of a coin flip. Twenty random starts all end there.
Now cheat. Press Add feature x₁·x₂. Give the neuron a third input, the product x₁x₂, which is 1 only at (1, 1). That corner lifts off the floor, and a flat plane can now slice the yes-points from the no-points. One exact solution is w = (1, 1, −2), b = −0.5: check (1, 1): 1 + 1 − 2 − 0.5 = −0.5 → no. Trained, the neuron gets all four right with loss 0.005103.
But you chose that feature. The real question, and the cliffhanger into Class 2, is how a network can invent such features by itself. The answer is to put neurons in front of the neuron, a hidden layer, and send the gradient back through them with the same chain rule you just used. That's backpropagation.
What you built tonight
A neuron is a weighted sum plus a bias, z = w·x + b, passed through an activation. Geometrically it draws one line: w's direction picks the line, |b|/‖w‖ places it, and ‖w‖ sets the confidence. A step can't learn because its slope is zero. The sigmoid can, and with cross-entropy its gradient collapses to (a − y)x, error times input. Repeat that one step and it trains AND, OR and NAND, but no amount of it can train XOR. Next class adds the hidden layer, derives backpropagation through it, and studies the loss landscape and the learning rate that gradient descent walks.
- Nielsen, Neural Networks and Deep Learning, ch. 1: sections "Perceptrons", "Sigmoid neurons", "Learning with gradient descent". ~45 min · core
- 3Blue1Brown, "But what is a neural network?" (ch. 1) and "Gradient descent, how neural networks learn" (ch. 2). 18:40 + 20:33 · intuition
- McCulloch & Pitts (1943), "A logical calculus of the ideas immanent in nervous activity", Bull. Math. Biophysics 5(4):115–133. ~10 min · abstract (paywalled)
- Rosenblatt (1958), "The perceptron: A probabilistic model for information storage and organization in the brain", Psychological Review 65(6):386–408. ~15 min · paywalled
- Minsky & Papert, Perceptrons, expanded ed., MIT Press (1987; 1st ed. 1969). ~5 min · publisher page
- Rumelhart, Hinton & Williams (1986), "Learning representations by back-propagating errors", Nature 323:533–536. ~15 min · Class 2 preview
- Optional: Karpathy, "building micrograd" (2:25:52; watch the first ~50 min), with code.
Train this neuron yourself, with no frameworks and no autograd. Fill five TODOs in starter.py: sigmoid, forward, bce, grads (derive ∂L/∂z on paper first) and the update step. Then run the tests.
cd lab python3 starter.py --part 1 # z = 0.250000 a = 0.562177 python3 -m pytest -q tests # or: python3 tests/test_lab.py LAB_TARGET=solution python3 -m pytest -q tests
Parts: 1 forward pass by hand · 2 gradient check · 3 train AND · 4 train the blobs (accuracy 0.990) · 5 XOR fails at ln 2. Challenges in challenges.md: "MSE whispers", "find the cliff" (Class 2 preview) and "cheat XOR". AGENTS.md tells your coding agents to give hints, not answers.
Ten questions
Instant feedback. Your answers are saved in this browser (localStorage), so you can reload safely.