the perceptron: no thoughts, just dot products

we're on mars. we have two biological castes of martians:

our job: build a machine that looks at a martian's measurements, then flashes the right light.

we have a clipboard. we start with two measurements:

• $H$ = height in cm
• $W$ = weight in kg

we write our measurements as a vector, which, for now, just means the measurements stacked in a bracket:

on the scatter plots below, i’ll put weight on the horizontal axis and height on the vertical, because it reads cleaner. in the algebra, ${\xx \mathbf{x}}$ just means “the measurements”; the order is arbitrary as long as ${\ww \mathbf{w}}$ uses the same order.

before we do any math, solve the problem visually.

we want a decision boundary: a line that separates the chonkers ($+1$) from the skinny legends ($-1$).

try it yourself. drag the two handles to position the line so it separates the two groups.

now: what is that line, mathematically?

building the decision engine from scratch

how do we turn two numbers into one decision? we invent a score, then choose a cutoff: $\text{score} > c$ means chonk.

look back at the scatter you just dragged a line through. in this toy dataset, the chonkers mostly live to the right: heavier.

so a decent first bet is a score that rises when either measurement rises: move right, score goes up; move up, score goes up.

this score adds centimeters to kilograms like they're the same thing. we'll fix that. but it already collapses the clipboard to a single number we can threshold ("chonker if score > c"), and the threshold has a shape: the points where $\text{score} = c$ form a straight diagonal line. that's the thing you just drew, rediscovered as algebra.

two problems remain: cm and kg aren’t comparable, and sometimes one feature should count more than the other.

weights are exchange rates

if height should count double, you'd write:

that “2” is literally an exchange rate.

replace the hard-coded 1s with adjustable weights, $w_1$ and $w_2$:

interpretation:

• one cm of height buys you $w_1$ score points
• one kg of weight buys you $w_2$ score points

this lets us put heterogeneous measurements on the same scoreboard.

bias: pick the cutoff

we need one more number: a cutoff $c$. that’s the line in the sand.

• if $\text{score} > c$ → chonker
• if $\text{score} < c$ → skinny legend
• if $\text{score} = c$ → exactly on the boundary

what does that boundary look like? plot every point where $\text{score} = c$. you get a straight line: that's what you were dragging around.

that straight line is just what "tied score" looks like in 2D. with three features it's a plane. with a hundred it's a hyperplane.

comparing to $c$ is the same as moving $c$ to the left and comparing to zero. define $b = -c$ and write:

same boundary, cleaner sign check.

sanity check with real numbers

here's the equation you found:

test it on two martians:

• H=175, W=78 (a chonker):

• H=160, W=52 (a skinny legend):

the chonker should score above the cutoff and the skinny legend below it. if either verdict reads wrong, your line probably needs work: scroll up and drag until both come out right. (one caveat: the printed check rounds your coefficients, so a line that only barely squeaks past a point can round the wrong way. give the boundary some breathing room.) the machine has no judgment beyond the three numbers you picked.

three numbers → a machine that can tell them apart.

the general form

the weights $w_1, w_2$ control the angle of the boundary. the bias $b$ (or equivalently the cutoff $c = -b$) shifts it.

it works for any number of features. add age, antennas? just add more weighted terms. it's always a flat cut through the space (a "hyperplane").

we can fold the bias into the weights to keep the algebra clean: treat it as a weight $w_0$ attached to a constant input $x_0 = 1$. then:

which is a product of two vectors:

so if we define:

the whole machine is:

one vector of weights, one vector of inputs. multiply and sum. (that's all the raised $\top$ means: tip the column of weights onto its side so it can pair up with the inputs. wherever you see ${\ww \mathbf{w}}^\top{\xx \mathbf{x}}$, read "multiply matching entries, add them up.")

we have this equation ${\ww \mathbf{w}}^\top {\xx \mathbf{x}} = 0$. what does it look like?

ignore the bias for a second. so far a vector has been a column of numbers. plot those numbers as coordinates (first number along one axis, second along the other) and the same vector becomes an arrow from the origin to that point. one object, two costumes. the arrow's length gets written $\|{\xx \mathbf{x}}\|$, and it's just pythagoras on the column: $\|{\xx \mathbf{x}}\| = \sqrt{x_1^2 + x_2^2}$. that is all the double bars ever mean, here or anywhere below.

together, $(w_1, w_2)$ form a vector: an arrow pointing somewhere in height-weight space. that's where the machine is looking.

the score measures how well a point ${\xx \mathbf{x}}$ aligns with that direction:

where $\theta$ is the angle between the two arrows. if trig is a distant memory: $\cos \theta$ is a dial for agreement. it reads 1 when the arrows point the same way, slides to 0 as they approach 90°, and hits −1 when they're opposed.

• if they point in roughly the same direction ($\theta < 90^\circ$), the score is positive.
• if they point in opposite directions ($\theta > 90^\circ$), the score is negative.
• if they are exactly perpendicular ($\theta = 90^\circ$), the score is zero.

two notes. the dot in ${\ww \mathbf{w}} \cdot {\xx \mathbf{x}}$ is the same multiply-and-add as ${\ww \mathbf{w}}^\top{\xx \mathbf{x}}$: two notations, one operation. and the equality between multiply-and-add and lengths-times-agreement-dial is the one fact in this essay we'll assert without deriving.

the viz computes the score both ways, live. drag ${\ww \mathbf{w}}$ (where the machine looks) and ${\xx \mathbf{x}}$ (a martian) anywhere: the two numbers refuse to disagree, the score flips sign the instant the angle crosses 90°, and it grows when either arrow gets longer.

and that last case is the boundary. $\text{score} = 0$ means "perpendicular to ${\ww \mathbf{w}}$." every point perpendicular to the weight vector scores zero, and that set of points forms a line (in 2d), a plane (in 3d), a hyperplane (in general).

learning is just rotating the flashlight so that the beam illuminates the chonkers and leaves the skinny legends in the dark.

one loose end: a real objection, though if 3D slicing makes your head hurt, skip this paragraph freely; nothing below depends on it. the line you dragged in 1.2 didn't pass through the origin, but ${\ww \mathbf{w}}^\top{\xx \mathbf{x}} = 0$ is a wall through the origin (${\xx \mathbf{x}} = \mathbf{0}$ always scores zero, no matter what ${\ww \mathbf{w}}$ is). both are true, and the bias trick from 1.2 is what reconciles them. in the augmented space $[1, H, W]$, the boundary really is a flat wall through the origin, perpendicular to the full three-component ${\ww \mathbf{w}}$. but every actual martian lives on the slice $x_0 = 1$, one unit out along the bias axis. slice a tilted wall-through-the-origin with that sheet and you get an offset line. the bias $w_0$ controls the wall's tilt, and the tilt controls where the line lands in your 2D plot. so "ignore the bias" gave you a flat shadow of the honest picture: one dimension up, the flashlight geometry is exact.

the weights are the classifier. change the weights, you rotate the boundary.

we start with random weights and watch the first mistake happen.

our machine boots up. the knobs are set randomly:

meaning:

• bias = 5 (starts chonk-leaning)
• height coefficient = 0.1 (taller → slightly more chonk-leaning)
• weight coefficient = -0.3 (heavier → MORE skinny-leaning??? backwards, but it's random)

a martian waddles in:

• ground truth: this is a CHONKER (+1)
• measurements: height = 170 cm, weight = 76 kg

the machine computes:

step by step:

• bias term: $5 \times 1 = 5$
• height term: $0.1 \times 170 = 17$
• weight term: $-0.3 \times 76 = -22.8$

score is negative. machine predicts skinny legend (-1).

but this martian is a CHONKER.

false negative.

geometrically: ${\ww \mathbf{w}}^\top{\xx \mathbf{x}} < 0$, so this point is on the “negative” side of the boundary: behind the beam.

and it's not just our random-knob machine that makes mistakes. below is the line you found in 1.2, against the same crowd plus two new arrivals: a water-retaining chonker who measures skinny, and a dehydrated skinny legend who scans chonk. check the verdict under the plot. odds are your line misses someone too. mistakes happen even to good lines. the question is what to do after one.

our machine called a chonker a skinny legend. the beam is pointing the wrong way. we need to change the weights.

our incident chonker ($y=+1$) got a score of -0.8. multiply them together and you get -0.8. when the prediction and label disagree, $y \cdot \text{score}$ is always negative.

take one training example $({\xx \mathbf{x}}, y)$, where the label $y$ is either $+1$ (chonker) or $-1$ (skinny legend).

• score = ${\ww \mathbf{w}}^\top {\xx \mathbf{x}}$
• prediction = $\text{sign}(\text{score})$, the score's sign: $+1$ if positive, $-1$ if negative

when the prediction and the label agree, $y \cdot \text{score}$ is positive (two positives or two negatives). when they disagree, it's negative:

what change to ${\ww \mathbf{w}}$ makes $y({\ww \mathbf{w}}^\top{\xx \mathbf{x}})$ go up the fastest?

suppose we add some $\Delta\mathbf{w}$:

the first term is the old mistake. the second term is the only knob we can turn.

the score is just a simple sum of multiplications. if you nudge one weight by some amount $\Delta w_i$, the overall score shifts by exactly that amount multiplied by the input $x_i$.

so the example itself tells you where the leverage is. for our incident ${\xx \mathbf{x}}=[1,170,76]$:

• add 1 to $w_0$ → score changes by 1
• add 1 to $w_1$ → score changes by 170
• add 1 to $w_2$ → score changes by 76

so we want $\Delta\mathbf{w}$ to have a big positive dot product with $y{\xx \mathbf{x}}$.

if you hold the step size fixed (say, $\|\Delta\mathbf{w}\| = 1$), the dot product $\Delta\mathbf{w}^\top (y{\xx \mathbf{x}}) = \|\Delta\mathbf{w}\|\,\|{\xx \mathbf{x}}\|\cos\theta$ is maximized at $\theta = 0$, when $\Delta\mathbf{w}$ points in the same direction as $y{\xx \mathbf{x}}$. just like the flashlight beam in 1.3, we want to physically rotate the weights to point exactly at the missed example.

point $\Delta\mathbf{w}$ directly at $y{\xx \mathbf{x}}$, scaled by a positive number $\eta$ that you choose (the learning rate: how big a step to take).

when $y=+1$ you add ${\xx \mathbf{x}}$; when $y=-1$ you subtract ${\xx \mathbf{x}}$. same rule.

that’s the update: when you’re wrong, move toward the labeled point.

in pseudocode:

initialize w (zeros are fine)
repeat for epochs:                      # one epoch = one pass through all points
  for (x, y) in data:
    pred = +1 if dot(w, x) > 0 else -1  # tie (score exactly 0) breaks to -1
    if pred != y:                       # mistake
      w = w + η * y * x

the tie-break is load-bearing. start from ${\ww \mathbf{w}} = \mathbf{0}$ and every score is exactly zero. if a tied score counted as correct, the loop would never fire a single update; the machine would sit at zero forever, technically never wrong by its own generous accounting. calling score 0 a $-1$ means the first chonker who walks in registers as a mistake and gets the weights moving.

here's one update, before and after: the boundary rotates toward the missed point.

the disagreement detector from 1.5 deserves a name. the signed margin is the score times the label:

• $m > 0$: correct side
• $m < 0$: wrong side (update fires)
• $m = 0$: on the fence; the tie-break from 1.5 calls it $-1$, so it's a mistake exactly when the martian is a chonker

either way, an update only ever fires when $m \le 0$. file that fact away; it carries the whole proof in 1.7.

one small gear before the algebra: a vector dotted with itself is its own squared length, ${\xx \mathbf{x}}^\top{\xx \mathbf{x}} = \|{\xx \mathbf{x}}\|^2$. multiply matching entries with themselves and you've built pythagoras's sum of squares.

after an update ${\ww \mathbf{w}} \leftarrow {\ww \mathbf{w}} + \eta y {\xx \mathbf{x}}$:

since $\eta > 0$ and $\|{\xx \mathbf{x}}\|^2 > 0$ for any real input, the signed margin strictly increases on that example.

step through it yourself below. each press visits one point: correct ones pass quietly, wrong ones fire the update and the boundary lurches. nothing protects the other points while it lurches.

one update makes you less wrong on that point. it doesn't promise the whole dataset stops.

can the dataset keep forcing updates forever?

if the data is separable, the perceptron runs out of mistakes. the proof tracks two quantities on a collision course.

(in 1.4 we booted up with random knobs because it makes a better story. the proof starts from zero because it makes cleaner algebra; a nonzero start survives the same argument, it just tacks a constant onto each bound and shifts the final count.)

the mandate is almost the definition of margin. each update adds $\eta y{\xx \mathbf{x}}$, and $y(\mathbf{u}^\top{\xx \mathbf{x}}) \ge {\gm \gamma}$ for every training point; that's what "classifies everything with margin ${\gm \gamma}$" means. so each mistake pushes ${\ww \mathbf{w}}^\top\mathbf{u}$ up by at least $\eta{\gm \gamma}$, and starting from zero, $k$ mistakes force ${\ww \mathbf{w}}^\top\mathbf{u} \ge k\eta{\gm \gamma}$.

the budget is the claim that should bother you. vector lengths don't behave this politely in general: add $k$ aligned vectors and the length grows linearly, not like $\sqrt{k}$. expand what one update does to the squared length:

(that expansion is school's $(a+b)^2 = a^2 + 2ab + b^2$, with dot products standing in for multiplication.)

stare at the middle term. it's the signed margin from 1.6, scaled by $2\eta$. and updates only fire on mistakes, when $y({\ww \mathbf{w}}^\top{\xx \mathbf{x}}) \le 0$. the middle term is never positive at the moment we add to ${\ww \mathbf{w}}$. the trigger that causes the growth is the same condition that caps it: the perceptron only ever lengthens its weights when the weights were pointing the wrong way, and adding a vector you disagree with mostly cancels.

so each mistake adds at most $\eta^2\|{\xx \mathbf{x}}\|^2 \le (\eta {\rr R})^2$ of squared length. after $k$ mistakes, starting from zero: $\|{\ww \mathbf{w}}\|^2 \le k(\eta {\rr R})^2$.

watch it happen with numbers. say ${\ww \mathbf{w}} = [1, 2]$ meets ${\xx \mathbf{x}} = [1, -2]$ with label $y = +1$ and $\eta = 1$: the score is $1 - 4 = -3$, a mistake, so we add ${\xx \mathbf{x}}$. the expansion predicts the new squared length: $5 + 2(-3) + 5 = 4$. check it directly: ${\ww \mathbf{w}} + {\xx \mathbf{x}} = [2, 0]$, squared length $4$. we added an arrow of squared length 5 and the weights got shorter. that's the middle term doing its job.

the same fact, draggable. the shaded half is where mistakes live: every point there scores negative against ${\ww \mathbf{w}}$, so it is the only region the update rule ever adds from. drag ${\xx \mathbf{x}}$ around both halves and keep your eye on the middle term. in the shaded half it is never positive, and $\|{\ww \mathbf{w}'}\|^2$ stays under the naive sum. in the unshaded half the middle term turns positive and the lengths would stack, but the perceptron never adds from there.

the race is plotted below. ${\gm \gamma}$ and ${\rr R}$ set the slopes (the sliders wear the same colors); drag $k$ forward and watch the linear mandate close in on the square-root ceiling.

• the mandate (dot product) demands linear growth ($k$).
• the budget (length) only allows square-root growth ($\sqrt{k}$).

but ${\ww \mathbf{w}}^\top \mathbf{u} \le \|{\ww \mathbf{w}}\|\,\|\mathbf{u}\| = \|{\ww \mathbf{w}}\|$ (the formal name for this ceiling is the cauchy-schwarz inequality, but geometrically it just means what we saw in 1.3: $\cos \theta$ maxes out at 1). a vector's projection onto a unit vector (its shadow along that direction) can never be longer than the vector itself.

so:

a linear function of $k$ will always eventually overtake a square root of $k$. but a projection can never outgrow the vector itself. so the only escape is that $k$ stops growing: the algorithm must run out of mistakes.

the collision happens when the floor meets the ceiling:

divide by $\eta \sqrt{k}$:

square both sides:

the whole proof fits in one breath:

read it left to right: the left end rises like $k$, the right end rises like $\sqrt{k}$, and the chain has to hold after every single mistake. a straight line cannot stay under a square root forever. so something must give, and the only thing that can give is $k$. squeeze the ends together and out falls $k \le ({\rr R}/{\gm \gamma})^2$.

look at what survived the algebra. the learning rate didn't: it divided out and never came back. with zero starting weights, every weight vector the perceptron visits is a scaled copy of the $\eta = 1$ run, and scaling never flips the sign of a score, so $\eta$ changes nothing about which mistakes happen. the knob everyone tunes first is, for this algorithm, decorative. the dimension count never even appeared: the bound sees the data only through ${\rr R}$ and ${\gm \gamma}$. two measurements per martian or two million: same geometry, same budget.

the bound is a number you can compute from the data before training starts. the loop can keep running forever, but the updates run out: one more mistake would push the mandate through the cauchy-schwarz ceiling, and that inequality doesn't bend. the machine stops misclassifying martians. not "probably won't keep failing." cannot.

now watch it happen. the trainer below starts from zero weights. before you press run, make a prediction: does the boundary glide into place, or thrash before it settles?

run the learn mission to zero mistakes. every lurch of the boundary is one update, and the theorem you just read is the reason the lurching ends. then run fail. that dataset has one flipped label, so no golden $\mathbf{u}$ exists, there's no ${\gm \gamma} > 0$, the mandate evaporates, and the thrashing never has to stop. that's when we reach for the pocket algorithm (keep the best weights seen so far) or switch to a soft-margin classifier like an SVM.

exercise 4 has you run the race yourself: four martians, a pencil, both bounds checked at every mistake.

the convergence proof says the perceptron must stop. it doesn't say it stops gracefully.

remember our incident martian from 1.4? ${\xx \mathbf{x}} = [1,\, 170,\, 76]$. with $\eta = 1$, the margin jump on each update is $\eta\|{\xx \mathbf{x}}\|^2 = 1^2 + 170^2 + 76^2 = 34{,}677$. one mistake and the weights are in orbit.

why so violent? because the inputs are raw units. against a bias input of 1, the measurements are a hundred times louder, and the squared length stacks the squares: height alone contributes $170^2 = 28{,}900$ of the 34,677. the convergence bound is $({\rr R}/{\gm \gamma})^2$, and it's the ratio that matters: multiply every coordinate by 10 and ${\gm \gamma}$ inflates right alongside ${\rr R}$, changing nothing. the damage comes from features that are loud relative to each other: one booming coordinate inflates the radius while the boundary still has to be negotiated by the quiet ones, so ${\rr R}/{\gm \gamma}$ balloons.

the fix is boring and universal: standardize features. subtract each feature's average $\mu_i$ and divide by its typical spread $\sigma_i$ (the standard deviation), $x_i \leftarrow (x_i - \mu_i)/\sigma_i$, so no single dimension dominates. after standardization, our martian's coordinates are order-1 instead of order-100, the updates stay sane, and the convergence bound tightens.

you can feel this one too: the trainer in 1.7 has a scale mission. run it raw, open the advanced panel, and watch $\|\Delta\mathbf{w}\|$ in the telemetry: same $\eta$, violent steps. flip to standardized and the steps shrink to something civilized.

the recipe

• standardize features: $x_i \leftarrow (x_i - \mu_i)/\sigma_i$ (save $\mu_i,\sigma_i$).
• add bias: include $x_0 = 1$ so the boundary can shift.
• shuffle each epoch: same points, fewer dumb cycles.
• early stop: stop when mistakes hit 0 (separable) or validation stops improving (non-separable).
• average or pocket: when in doubt, ship ${\ww \bar{\mathbf{w}}}$ or the best ${\ww \mathbf{w}}$ seen.

what if the data can't be separated by any line?

four points at the corners of a square. label them by diagonal: one diagonal pair is +1, the other is -1.

try to draw a line that separates them. really, drag it around.

you can't. sweep the boundary through every angle: at least one side always ends up mixed, and the counter never reaches zero. (it's even being generous: it scores whichever side-labeling flatters your line.) the perceptron will oscillate forever, shoving the boundary back and forth, never settling.

two lines prove it. both diagonals share the same midpoint: the center of the square. and a half-plane has no dents (the formal word is convex): if one side of a line holds a segment's two endpoints, it holds everything between them. so the side holding both +1s would own the center, and the side holding both -1s would own the center too. one point, claimed by both sides of a line. no.

the score $w_1 A + w_2 B$ sees each input's contribution independently: it knows "A is big" and "B is big" as separate facts.

what it can't see is whether A and B agree. agreement lives between the inputs, and a weighted sum has no term for between: every input contributes on its own, blind to the others.

there are two ways out:

1. change the representation (feature engineering)
2. learn the representation (a neural network)

if the flashlight can't see the pattern, give it a new dimension to look along: one that turns "agreement," the exact thing the weighted sum is blind to, into a raw number we can hand over as an input.

look at the inputs. they are 0s and 1s. we need to compute something new from them. and "new" is a real constraint, because most of the obvious candidates aren't:

• $A+B$ is a weighted sum. the machine builds weighted sums: hand it $A+B$ as a feature and you've given it a knob it could already turn by setting $w_1 = w_2$. no new direction. same flatland, extra steps.
• $A-B$, $3A + 2B$, any recipe of the form $\alpha A + \beta B$: same problem. they're all inside the flashlight's reach already.

the new input has to be something no weighted sum can fake. multiplication is the cheapest such thing: $A \cdot B$ is 1 exactly when both inputs are on, and no $\alpha A + \beta B + \gamma$ reproduces that table: matching the three 0s forces $\alpha = \beta = \gamma = 0$, which then can't produce the 1.

by manually computing $A \cdot B$ and feeding it in alongside the original inputs, we give the machine a dedicated knob for that relationship. geometrically, the new coordinate lifts the (1,1) corner out of the plane, and now a flat sheet of glass can separate the classes. the sheet has to tilt, angled so the lifted (1,1) and the ground-floor (0,0) both end up above it while the two +1 corners sit below. drag the slider and watch the corner climb until the glass can finally get between the classes.

this preprocessing step is called a feature map:

(four coordinates once you count the bias; the picture above shows the three that move.)

the mapped points are:

one separating weight vector is ${\ww \mathbf{w}} = [-1, 2, 2, -4]$. check:

• $(0,0)$: $-1 + 0 + 0 + 0 = -1$ → -1 ✓
• $(0,1)$: $-1 + 0 + 2 + 0 = 1$ → +1 ✓
• $(1,0)$: $-1 + 2 + 0 + 0 = 1$ → +1 ✓
• $(1,1)$: $-1 + 2 + 2 - 4 = -1$ → -1 ✓

translation: “if BOTH inputs are 1, subtract 4 from the score.” that feature makes XOR separable.

but this is the catch: you had to know to add $A\cdot B$. the perceptron can't see features it wasn't given: $A \cdot B$ was in the beam's blind spot. it can only learn weights for the dimensions you hand it.

that $A\cdot B$ term is the simplest interaction feature: it turns “both are on” into a single coordinate.

in the martian story, $H\cdot W$ is the same move. continuous measurements cost it the clean on/off behavior of $A\cdot B$, but the knob still depends on both measurements jointly, which no weighted sum of $H$ and $W$ alone can express.

the feature map worked, but only because we knew $A\cdot B$ was the right feature.

for XOR that's trivial: two inputs, one pairwise interaction to try. for our martians, we could guess $H\cdot W$. but for a real problem with a hundred measurements, there are $4{,}950$ possible pairwise interactions. three-way combinations? $161{,}700$. you're not hand-crafting your way out of that.

the perceptron can learn any boundary in the space you give it. but it can't see past that space. if the feature that matters is a combination you didn't think to include, no amount of weight-nudging will find it. the flashlight can only illuminate dimensions that exist.

what if the machine could build its own dimensions?

that's the idea behind a neural network. instead of you designing $A\cdot B$ and hoping, you give the machine a stack of layers. each layer computes weighted sums (exactly like a perceptron), then bends the result through a nonlinearity (like clamping negatives to zero). the bend is load-bearing: stack linear layers without it and the tower collapses, because a weighted sum of weighted sums is just one big weighted sum, and we spent this whole section proving what a single one can't do. with the bend, the early layers combine raw inputs into new features, and the later layers combine those into higher-order ones.

nobody tells the network about $A\cdot B$. training rewards whatever accidentally behaves like $A\cdot B$, so that behavior grows.

the final layer is still a dot product. everything we built in this essay (the score, the boundary, the sign check) is still there at the output. a neural network is a perceptron whose inputs are learned features rather than raw measurements.

but stacking layers creates a problem the perceptron never had.

in a single perceptron, when the machine miscalls a chonker, we know exactly who to blame: the one weight vector ${\ww \mathbf{w}}$. we nudge it toward $y{\xx \mathbf{x}}$ and move on. one knob, one gradient, one step.

in a network with three layers and thousands of weights, the output depends on all of them. when the prediction is wrong, the blame is smeared: maybe an early layer built a bad feature, maybe the last layer drew a bad boundary through good features, most likely some of both. every weight had a hand in the output, and each one deserves a different share of the blame.

credit assignment: which weight to blame, by how much. that's the next essay.

1. the machine: weighted sum of inputs → is it positive?
2. the problem: we don't know the right weights
3. the fix: when wrong, ${\ww \mathbf{w}} \leftarrow {\ww \mathbf{w}} + \eta y {\xx \mathbf{x}}$
4. why it helps: margin increases by $\eta \|{\xx \mathbf{x}}\|^2$ on the mistake example
5. the catch: if a perfect line with breathing room ${\gm \gamma}$ exists, the total number of mistakes is capped at $({\rr R}/{\gm \gamma})^2$, where ${\rr R}$ is the length of the longest input vector. if no such line exists, the updates can thrash forever
6. the deeper catch: if your representation is wrong (xor), no amount of linear sweat will save you
7. the open question: neural nets learn their own features, but when the output is wrong, which weights do you blame? (next essay: backpropagation)

the goal: you should be able to do these with a pencil, and you should know you're right at the end.

w = [0, 0, 0]
w_sum = [0, 0, 0]; t = 0
w_pocket = w; best_mistakes = count_mistakes(w, all_data)
 
for epoch in 1..N:          # one pass through all points
  # (shuffle in production; fixed order here for hand-tracing)
  mistakes = 0
  for (x, y) in data:
    score = dot(w, x)
    pred = +1 if score > 0 else -1
    if pred != y:
      w = w + eta * y * x
      mistakes += 1
      m = count_mistakes(w, all_data)   # pocket check; w only changed here
      if m < best_mistakes:
        w_pocket = w; best_mistakes = m
    w_sum += w; t += 1
 
  if mistakes == 0: break
 
w_avg = w_sum / t
return w, w_pocket, w_avg