How Neural Networks Learn

The Goal: Minimise the Loss

A neural network starts with random weights. It makes predictions — mostly wrong at first. The training process is simply: measure how wrong the predictions are, then adjust the weights to make them less wrong. Repeat thousands of times.

The "measure of wrongness" is the loss function. It takes the network's prediction and the correct answer, and returns a single number — higher means more wrong, lower means closer to correct. Training is the process of minimising this number.

Loss Functions

Categorical Cross-Entropy (Classification)

Used when the output is a probability distribution over multiple classes (like Softmax outputs). It measures how different the predicted probability distribution is from the true label.

L = −Σ yᵢ · log(ŷᵢ)

If the correct class is "5" and the model outputs 95% confidence for "5", the loss is very low. If it outputs 5% for "5", the loss is very high (log of a small number is a large negative number — the minus sign makes it positive).

Why log? The log function penalises confident wrong answers much more than uncertain wrong answers. If the model says "I'm 99% sure this is a 3" and it's actually a 7, the loss is catastrophically high. This is exactly what you want — punish overconfident mistakes severely to force the model to be well-calibrated.

Mean Squared Error (Regression)

Used when predicting a continuous value. Squares the difference between prediction and truth, then averages across all examples. Not used in the SVHN project (classification, not regression), but fundamental to understand.

MSE = (1/n) · Σ (yᵢ − ŷᵢ)²

Gradient Descent

The loss function creates a landscape in weight space — a high-dimensional surface where lower points mean better predictions. Training is navigating this landscape downhill toward a minimum.

Gradient descent computes the direction of steepest ascent at the current position (the gradient), then moves in the opposite direction (downhill) by a small step.

w_new = w_old − η · ∂L/∂w

Where η (eta) is the learning rate — how large a step to take. ∂L/∂w is the gradient of the loss with respect to each weight.

The Learning Rate Dilemma

Too high vs. too low Too high: The step overshoots the minimum. The loss bounces around or diverges — training becomes unstable.
Too low: Training converges, but extremely slowly. You need 10× more epochs to reach the same result.
Typical starting values: 0.001 (Adam) or 0.01 (SGD). Always validate via training curves.

Mini-Batch Gradient Descent

Computing the gradient over the entire dataset before each update (batch gradient descent) is accurate but slow. Computing it on a single example (stochastic gradient descent) is fast but noisy. The standard compromise: mini-batches — compute the gradient on a random subset (typically 32 or 64 examples), update, repeat.

Backpropagation

After the forward pass (computing the prediction and loss), the network needs to know: which weights contributed most to the error, and in which direction should they move to reduce it?

Backpropagation is the algorithm that computes these gradients efficiently using the chain rule of calculus. It starts at the output layer (where the loss is measured), computes each layer's gradient, and "propagates" the signal backwards through the network to the input.

The chain rule, intuitively If layer 3's output affected layer 4, and layer 4's output affected the loss, then layer 3's weights affected the loss through layer 4. The chain rule multiplies the gradients along this path. Backprop automates this for every weight in the network simultaneously — a computation that would be impossibly tedious by hand for millions of parameters.

The Full Training Loop

1
Forward pass. Feed a mini-batch of examples through the network. Each layer computes its output. The final layer produces predictions.
2
Compute loss. Compare predictions to true labels using the loss function. One number summarises how wrong the batch was.
3
Backward pass (backpropagation). Compute the gradient of the loss with respect to every weight in the network.
4
Update weights. Move each weight slightly in the direction that reduces the loss (gradient descent step).
5
Repeat for every mini-batch. One full pass through all training data = one epoch. Train for dozens to hundreds of epochs.

SVHN training loop summary:

Optimiser:    Adam (lr=0.001)
Batch size:   64
Epochs:       50 (with early stopping)
Loss:         Categorical cross-entropy

Epoch 1:  train_loss=2.30  val_acc=18%    ← random-ish weights
Epoch 5:  train_loss=1.45  val_acc=52%    ← learning fast
Epoch 20: train_loss=0.38  val_acc=85%    ← slowing down
Epoch 35: train_loss=0.21  val_acc=91%    ← converging
Epoch 42: train_loss=0.18  val_acc=92.2%  ← best val accuracy

The Adam Optimiser

Plain gradient descent uses the same learning rate for every weight. Adam (Adaptive Moment Estimation) adapts the learning rate individually for each weight based on the history of its gradients. Weights that have been updated consistently get smaller steps (they're likely near a minimum); weights with erratic gradients get larger steps.

Adam combines two ideas: momentum (remembering recent gradient directions to smooth out noise) and adaptive learning rates (per-weight step sizes). In practice, it converges faster and more reliably than plain SGD for most architectures.

Practical rule Default to Adam with lr=0.001. If the model isn't converging, lower the learning rate. If training is very slow, try a higher rate. Add a learning rate scheduler (reduce on plateau) for fine-tuning.

Interactive: Training Curves

Watch loss decrease and accuracy climb over 50 epochs. The gap between training and validation reveals how much the model is overfitting. Toggle between Loss and Accuracy views.

CNN2 Training History · 50 Epochs

Training Validation

Epoch 50/50 — Press ▶ Play to animate from epoch 1

In the SVHN Project

All four models (ANN1, ANN2, CNN1, CNN2) used the Adam optimiser with categorical cross-entropy loss. Training curves showed classic learning behaviour: rapid improvement in early epochs, then gradual convergence. Early stopping (patience=5) halted training when validation loss stopped improving, preventing wasted compute and overfitting. The jump from ANN2 (77%) to CNN1 (87%) demonstrated that architecture change — not just more training — was responsible for the accuracy gain. See Chapter 4 — Training →

← Activation Functions Next: Overfitting →