Overfitting & Regularisation

The Generalisation Problem

The goal of a neural network isn't to perform well on training data — it's to perform well on new data it hasn't seen before. A model that achieves 99% accuracy on training images but 60% on new images has learned nothing useful. It has memorised the training set.

This gap between training performance and real-world performance is overfitting. It's the central challenge of machine learning.

Why overfitting happens Neural networks have millions of parameters — more than enough capacity to memorise every training example, including the noise, quirks, and irrelevant patterns specific to that dataset. A model that memorises rather than learns won't generalise. The training loss keeps falling; the validation loss starts rising. This divergence is the signature of overfitting.

Underfitting, Good Fit, Overfitting

Underfitting

The model is too simple to capture the pattern. Both training and validation accuracy are low. The model hasn't learned enough — add capacity (more layers/neurons) or train longer.

Good Fit

Training and validation accuracy are both high and close to each other. The model has learned the underlying pattern without memorising noise. This is the target.

Overfitting

Training accuracy is high; validation accuracy is noticeably lower. The gap grows over time as training continues. The model has started memorising training-specific noise.

How to detect it

Plot training vs. validation loss over epochs. If they diverge — training keeps falling while validation plateaus or rises — you're overfitting. Always monitor both.

Training vs Validation Loss — Interactive

Train loss

Val loss

Select a scenario below Switch between the three regimes to see exactly what they look like on a real loss curve.

Dropout

Dropout is a regularisation technique that randomly "switches off" a fraction of neurons during each training step. A neuron that's dropped out produces no output and receives no gradient update for that step.

With a dropout rate of 0.5, each neuron has a 50% chance of being ignored during any given training pass. This sounds destructive — but it's extraordinarily effective.

Why it works

Dropout forces the network to learn redundant representations. No single neuron can be relied upon — it might be dropped at any moment. So multiple neurons must learn to represent the same feature independently. The result is a network that's robust to the absence of individual neurons.

It also acts as training an ensemble of many different networks simultaneously. Each training step uses a different random subset of neurons — a different "sub-network." The final model is effectively an average of all these sub-networks, which is much more generalised than any single one.

Dropout at inference time During testing and prediction, dropout is turned OFF. All neurons are active. But their outputs are scaled down by the dropout probability — so the expected output is the same as during training. Keras/TensorFlow handles this automatically.

Typical dropout placement in SVHN models:

Dense(512) → LeakyReLU → Dropout(0.4)
Dense(256) → LeakyReLU → Dropout(0.3)
Dense(10)  → Softmax

Dropout rate decreases in later layers — earlier layers
need more regularisation (more parameters, higher risk
of memorising specific features).

Batch Normalisation

Batch Normalisation (BatchNorm) normalises the output of each layer to have zero mean and unit variance — computed across the current mini-batch. Then it applies learnable scale and shift parameters, letting the network adjust the normalisation if needed.

For each layer output x in a batch:
  μ = mean(x)
  σ² = variance(x)
  x_norm = (x − μ) / √(σ² + ε)     ← normalise
  output = γ · x_norm + β            ← scale and shift (learnable)

What it solves

Internal covariate shift: As earlier layers update their weights, the distribution of inputs to later layers keeps changing. BatchNorm stabilises this — later layers always receive input with the same scale.
Allows higher learning rates: Without BatchNorm, high learning rates cause exploding or vanishing gradients. BatchNorm keeps activations in a stable range, making training more aggressive and faster.
Mild regularisation effect: Because normalisation is computed per-batch (which varies), it introduces slight noise — acting as a weak form of regularisation on top of its primary role.

BatchNorm placement Typically placed after the linear transformation and before the activation function: Dense → BatchNorm → ReLU. Some practitioners place it after the activation — empirically, either can work, and it's worth experimenting with your specific architecture.

Other Regularisation Techniques

Early Stopping

Monitor validation loss during training. When it stops improving for a specified number of epochs (the "patience"), halt training and restore the weights from the best epoch. Simple and extremely effective — it prevents the network from memorising noise in later epochs when training loss is still falling.

Data Augmentation

Artificially increase the diversity of training data by applying random transformations: flips, rotations, brightness changes, small crops. The model sees more variation without collecting new data. Highly effective for image tasks.

L2 Regularisation (Weight Decay)

Add a penalty to the loss function proportional to the square of each weight's magnitude. Large weights are penalised, pushing the model toward smaller, more evenly distributed weights — less likely to overfit to specific training examples.

In the SVHN Project

All four models used Dropout (rates 0.3–0.5) and Batch Normalisation after each dense layer. Early stopping with patience=5 halted training when validation loss plateaued. The effect was measurable: ANN2 without these techniques showed training accuracy of ~85% but validation accuracy of 77% — a 8-point gap signalling overfitting. With regularisation, the CNN2 final model showed a training/validation gap of under 2 points at 92.22% validation accuracy. See Chapter 5 — Regularisation →

← How Networks Learn Next: Convolutional Networks →