the perceptron: no thoughts, just dot products

we're on mars. we have two biological castes of martians:

our job: build a machine that looks at a martian's measurements, then flashes the right light.

we have a clipboard. we start with two measurements:

• $H$ = height in cm
• $W$ = weight in kg

we write our measurements as a vector:

on the scatter plots below, i’ll put weight on the horizontal axis and height on the vertical, because it reads cleaner. in the algebra, $\mathbf{x}$ just means “the measurements” — the order is arbitrary as long as $\mathbf{w}$ uses the same order.

before we do any math, solve the problem visually.

we want a decision boundary—a line that separates the chonkers ($+1$) from the skinny legends ($-1$).

try it yourself. drag the handles on the axes to position the line so it separates the two groups.

now: what is that line, mathematically?

building the decision engine from scratch

how do we turn two numbers into one decision? we invent a score, then choose a cutoff: $\text{score} > c$ means chonk.

look at the scatter. in this toy dataset, the chonkers mostly live to the right: heavier.

so a decent first bet is a score that rises when either measurement rises—move right, score goes up; move up, score goes up.

this score ignores units (cm vs kg). but it buys us two crucial things immediately:

1. a single scalar we can threshold ("chonker if score > c")
2. a clean geometry: the decision boundary $\text{score}=c$ becomes a straight line (a diagonal cut between bottom-left and top-right)

now we fix the problems: cm and kg aren’t comparable, and sometimes one feature should count more than the other.

weights are exchange rates

if height should count double, you'd write:

that “2” is literally an exchange rate.

replace the hard-coded 1s with adjustable weights, $w_1$ and $w_2$:

interpretation:

• one cm of height buys you $w_1$ score points
• one kg of weight buys you $w_2$ score points

this lets us put heterogeneous measurements on the same scoreboard.

bias: pick the cutoff

we need one more number: a cutoff $c$. that’s the line in the sand.

• if $\text{score} > c$ → chonker
• if $\text{score} < c$ → skinny legend
• if $\text{score} = c$ → exactly on the boundary

what does that boundary look like? plot every point where $\text{score} = c$. you get a straight line—that's what you were dragging around.

that straight line is just what "tied score" looks like in 2D. with three features it's a plane. with a hundred it's a hyperplane.

comparing to $c$ is the same as moving $c$ to the left and comparing to zero. define $b = -c$ and write:

same boundary, cleaner sign check.

sanity check with real numbers

here's the equation you found:

test it on two martians:

• H=175, W=78 (chonker): . score > cutoff → predicted chonker (+1).

• H=160, W=52 (skinny legend): . score < cutoff → predicted skinny legend (-1).

three numbers → a machine that can tell them apart.

the general form

the weights $w_1, w_2$ control the angle of the boundary. the bias $b$ (or equivalently the cutoff $c = -b$) shifts it.

it works for any number of features. add age, antennas? just add more weighted terms. it's always a flat cut through the space (a "hyperplane").

we can fold the bias into the weights to keep the algebra clean: treat it as a weight $w_0$ attached to a constant input $x_0 = 1$. then:

which is a product of two vectors:

so if we define:

the whole machine is:

one vector of weights, one vector of inputs. multiply and sum.

we have this equation ${\ww \mathbf{w}}^\top {\xx \mathbf{x}} = 0$. what does it look like?

ignore the bias for a second. together, $(w_1, w_2)$ form a vector — a direction in height-weight space. that's where the machine is looking.

the score measures how well a point $\mathbf{x}$ aligns with that direction:

where $\theta$ is the angle between them.

• if they point in roughly the same direction ($\theta < 90^\circ$), the score is positive.
• if they point in opposite directions ($\theta > 90^\circ$), the score is negative.
• if they are exactly perpendicular ($\theta = 90^\circ$), the score is zero.

and that last case is the boundary. $\text{score} = 0$ means "perpendicular to $\mathbf{w}$." every point perpendicular to the weight vector scores zero, and that set of points forms a line (in 2d), a plane (in 3d), a hyperplane (in general).

learning is just rotating the flashlight so that the beam illuminates the chonkers and leaves the skinny legends in the dark.

the weights are the classifier. change the weights, you rotate the boundary.

we start with random weights and watch the first mistake happen.

our machine boots up. the knobs are set randomly:

meaning:

• bias = 5 (starts chonk-leaning)
• height coefficient = 0.1 (taller → slightly more chonk-leaning)
• weight coefficient = -0.3 (heavier → MORE skinny-leaning??? backwards, but it's random)

a martian waddles in:

• ground truth: this is a CHONKER (+1)
• measurements: height = 170 cm, weight = 76 kg

the machine computes:

step by step:

• bias term: $5 \times 1 = 5$
• height term: $0.1 \times 170 = 17$
• weight term: $-0.3 \times 76 = -22.8$

score is negative. machine predicts skinny legend (-1).

but this martian is a CHONKER.

false negative.

geometrically: $\mathbf{w}^\top\mathbf{x} < 0$, so this point is on the “negative” side of the boundary — behind the beam.

our machine called a chonker a skinny legend. the beam is pointing the wrong way. we need to change the weights.

our incident chonker ($y=+1$) got a score of -0.8. multiply them together and you get -0.8. when the prediction and label disagree, $y \cdot \text{score}$ is always negative.

take one training example $(\mathbf{x}, y)$ where $y \in \{-1, +1\}$.

• score = ${\ww \mathbf{w}}^\top {\xx \mathbf{x}}$
• prediction = $\text{sign}(\text{score})$

when the prediction and the label agree, $y \cdot \text{score}$ is positive (two positives or two negatives). when they disagree, it's negative:

what change to $\mathbf{w}$ makes $y(\mathbf{w}^\top\mathbf{x})$ go up the fastest?

suppose we add some $\Delta\mathbf{w}$:

the first term is the old mistake. the second term is the only knob we can turn.

the score is just a simple sum of multiplications. if you nudge one weight by some amount $\Delta w_i$, the overall score shifts by exactly that amount multiplied by the input $x_i$.

so the example itself tells you where the leverage is. for our incident $\mathbf{x}=[1,170,76]$:

• add 1 to $w_0$ → score changes by 1
• add 1 to $w_1$ → score changes by 170
• add 1 to $w_2$ → score changes by 76

so we want $\Delta\mathbf{w}$ to have a big positive dot product with $y\mathbf{x}$.

if you hold the step size fixed (say, $\|\Delta\mathbf{w}\| = 1$), the dot product $\Delta\mathbf{w}^\top (y\mathbf{x}) = \|\Delta\mathbf{w}\|\,\|\mathbf{x}\|\cos\theta$ is maximized at $\theta = 0$ — when $\Delta\mathbf{w}$ points in the same direction as $y\mathbf{x}$. just like the flashlight beam in 1.3, we want to physically rotate the weights to point exactly at the missed example.

point $\Delta\mathbf{w}$ directly at $y\mathbf{x}$, scaled by $\eta$ (the learning rate).

when $y=+1$ you add $\mathbf{x}$; when $y=-1$ you subtract $\mathbf{x}$. same rule.

that’s the update: when you’re wrong, move toward the labeled point.

in pseudocode:

initialize w
repeat for epochs:          # one epoch = one pass through all points
  for (x, y) in data:
    if y * dot(w, x) < 0:    # mistake
      w = w + η * y * x

without this trick, we'd need two cases: "if $y=+1$, check that the score is positive; if $y=-1$, check that it's negative." multiplying by $y$ collapses both into one number — the signed margin — that should always be positive:

• $m > 0$: correct side
• $m < 0$: wrong side (update fires)

after an update ${\ww \mathbf{w}} \leftarrow {\ww \mathbf{w}} + \eta y {\xx \mathbf{x}}$:

since $\eta > 0$ and $\|{\xx \mathbf{x}}\|^2 > 0$ for any real input, the signed margin strictly increases on that example.

one update makes you less wrong on that point. it doesn't promise the whole dataset stops.

can the dataset keep forcing updates forever?

if the data is separable, the perceptron cannot loop forever. our machine sorting martians must eventually stop misclassifying them.

not "probably won't." cannot.

we can prove this by tracking two physical quantities that are on a collision course.

(drag the slider).

• the mandate (dot product) demands linear growth ($k$).
• the budget (length) only allows square-root growth ($\sqrt{k}$).

but $\mathbf{w}^\top \mathbf{u} \le \|\mathbf{w}\|\,\|\mathbf{u}\| = \|\mathbf{w}\|$ (the formal name for this ceiling is the cauchy-schwarz inequality, but geometrically it just means what we saw in 1.3: $\cos \theta$ maxes out at 1). a vector's projection onto a unit vector can never exceed the vector's own length.

so:

a linear function of $k$ will always eventually overtake a square root of $k$. but a projection can never outgrow the vector itself. so the only escape is that $k$ stops growing: the algorithm must run out of mistakes.

let's pin down the number. the "collision" happens when the lower bound hits the upper bound:

divide by $\eta \sqrt{k}$:

square both sides:

so the number of mistakes $k$ is bounded by $(R/\gamma)^2$. calculating for eternity is physically impossible.

if the data isn't separable (no such $\mathbf{u}$ exists), the mandate doesn't exist — there's no $\gamma > 0$ to force growth — and the perceptron can thrash forever. that's when we use the pocket algorithm (keep the best weights seen so far) or switch to a soft-margin classifier like an SVM.

the convergence proof says the perceptron must stop. it doesn't say it stops gracefully.

remember our incident martian from 1.4? $\mathbf{x} = [1,\, 170,\, 76]$. with $\eta = 1$, the margin jump on each update is $\eta\|{\xx \mathbf{x}}\|^2 = 1^2 + 170^2 + 76^2 = 34{,}677$. one mistake and the weights are in orbit.

why so violent? because height is in centimeters and weight is in kilograms. the height coordinate is roughly $100\times$ louder than everything else, so it dominates the squared norm. the convergence bound is $(R/\gamma)^2$ — if $R$ is inflated by one unscaled feature, the bound blows up quadratically.

the fix is boring and universal: standardize features ($x_i \leftarrow (x_i - \mu_i)/\sigma_i$) so no single dimension dominates. after standardization, our martian's coordinates are order-1 instead of order-100, the updates stay sane, and the convergence bound tightens.

the recipe

• standardize features: $x_i \leftarrow (x_i - \mu_i)/\sigma_i$ (save $\mu_i,\sigma_i$).
• add bias: include $x_0 = 1$ so the boundary can shift.
• shuffle each epoch: same points, fewer dumb cycles.
• early stop: stop when mistakes hit 0 (separable) or validation stops improving (non-separable).
• average or pocket: when in doubt, ship $\bar{\mathbf{w}}$ or the best $\mathbf{w}$ seen.

what if the data can't be separated by any line?

four points at the corners of a square. label them by diagonal: one diagonal pair is +1, the other is -1.

try to draw a line that separates them.

you can't. sweep the boundary through every angle — every orientation catches one +1 and one -1 on each side. the perceptron will oscillate forever, shoving the boundary back and forth, never settling.

a line can only cut the plane into two half-planes, and no half-plane contains exactly one diagonal of a square. the +1s and -1s alternate around the corners.

the score $w_1 A + w_2 B$ sees each input's contribution independently — it knows "A is big" and "B is big" as separate facts.

what it can't see is whether A and B agree. agreement between A and B isn't a direction — it's a relationship between directions, invisible to any weighted sum.

there are two ways out:

1. change the representation (feature engineering)
2. learn the representation (a neural network)

if the machine can't draw the boundary using A and B, we have to invent a new input that captures what it's missing.

the machine is looking at A and B as separate facts. what it can't see is whether they agree. we need to turn "agreement" into a raw number we can hand to it.

look at the inputs. they are 0s and 1s. how do you mathematically combine two 0s and 1s into a single number that screams "both are on"?

• you can't just add them ($A+B$). $1+0$ gives you 1, but so does $0+1$. the machine couldn't tell the difference between "only A is on" and "only B is on."
• you can't subtract them.

but if you multiply them ($A \cdot B$), you get a 1 if and only if both A and B are 1. every other combination yields 0.

by manually computing $A \cdot B$ and feeding it in alongside the original inputs, we give the machine a dedicated knob for that relationship. we literally pop the 2D plane into a 3D space, lifting the (1,1) corner up so a flat sheet of glass can slice cleanly underneath it.

this preprocessing step is called a feature map:

the mapped points are:

one separating weight vector is $\mathbf{w} = [-1, 2, 2, -4]$. check:

• $(0,0)$: $-1 + 0 + 0 + 0 = -1$ → -1 ✓
• $(0,1)$: $-1 + 0 + 2 + 0 = 1$ → +1 ✓
• $(1,0)$: $-1 + 2 + 0 + 0 = 1$ → +1 ✓
• $(1,1)$: $-1 + 2 + 2 - 4 = -1$ → -1 ✓

translation: “if BOTH inputs are 1, subtract 4 from the score.” that feature makes XOR separable.

but this is the catch: you had to know to add $A\cdot B$. the perceptron can't see features it wasn't given — $A \cdot B$ was in the beam's blind spot. it can only learn weights for the dimensions you hand it.

that $A\cdot B$ term is the simplest interaction feature: it turns “both are on” into a single coordinate.

in the martian story, $H\cdot W$ is the same move. it gives you a knob for “tall and heavy” that a purely linear score can’t express directly.

the feature map worked — but only because we knew $A\cdot B$ was the right feature.

for XOR that's trivial — two inputs, one pairwise interaction to try. for our martians, we could guess $H\cdot W$. but for a real problem with a hundred measurements, the pairwise interactions alone number $\binom{100}{2} = 4{,}950$. three-way? $161{,}700$. you're not hand-crafting your way out of that.

the perceptron can learn any boundary in the space you give it. but it can't see past that space. if the feature that matters is a combination you didn't think to include, no amount of weight-nudging will find it. the flashlight can only illuminate dimensions that exist.

what if the machine could build its own dimensions?

that's the idea behind a neural network. instead of you designing $A\cdot B$ and hoping, you give the machine a stack of layers. each layer computes weighted sums — exactly like a perceptron — then bends the result through a nonlinearity (like clamping negatives to zero). the early layers combine raw inputs into new features. the later layers combine those into higher-order ones.

the network doesn't know it needs $A\cdot B$. it builds something that works like $A\cdot B$ because the training signal rewards it. the feature map isn't designed. it's grown.

the final layer is still a dot product. everything we built in this essay — the score, the boundary, the sign check — is still there at the output. a neural network is a perceptron whose inputs are learned features rather than raw measurements.

but stacking layers creates a problem the perceptron never had.

in a single perceptron, when the machine miscalls a chonker, we know exactly who to blame: the one weight vector $\mathbf{w}$. we nudge it toward $y\mathbf{x}$ and move on. one knob, one gradient, one step.

in a network with three layers and thousands of weights, the output depends on all of them. when the prediction is wrong, which layer screwed up? the first layer, which built a bad feature? the second, which combined good features badly? the last, which drew the boundary in the wrong place? every weight had a hand in the output, and each one deserves a different share of the blame.

credit assignment — which weight to blame, by how much. that's the next essay.

1. the machine: weighted sum of inputs → is it positive?
2. the problem: we don't know the right weights
3. the fix: when wrong, ${\ww \mathbf{w}} \leftarrow {\ww \mathbf{w}} + \eta y {\xx \mathbf{x}}$
4. why it helps: margin increases by $\eta \|{\xx \mathbf{x}}\|^2$ on the mistake example
5. the catch: convergence needs separability (otherwise it can thrash)
6. the deeper catch: if your representation is wrong (xor), no amount of linear sweat will save you
7. the open question: neural nets learn their own features — but when the output is wrong, which weights do you blame? (next essay: backpropagation)

the goal: you should be able to do these with a pencil, and you should know you're right at the end.

w = [0, 0, 0]
w_sum = [0, 0, 0]; t = 0
w_pocket = w; best_mistakes = inf

for epoch in 1..N:          # one pass through all points
  # (shuffle in production; fixed order here for hand-tracing)
  mistakes = 0
  for (x, y) in data:
    score = dot(w, x)
    pred = +1 if score > 0 else -1
    if pred != y:
      w = w + eta * y * x
      mistakes += 1
    w_sum += w; t += 1

  total_mistakes = count_mistakes(w, all_data)
  if total_mistakes < best_mistakes:
    w_pocket = w; best_mistakes = total_mistakes
  if mistakes == 0: break

w_avg = w_sum / t
return w, w_pocket, w_avg