1. The Biological Inspiration
The human brain contains roughly 86 billion neurons. Each neuron receives electrical signals through branching structures called dendrites, integrates those signals in its cell body, and if the total input exceeds a threshold, fires an output signal down its axon to the next neuron. Information is encoded not in any single neuron, but in the patterns of activation across billions of them.
Artificial neural networks were designed to mimic this structure — not biologically faithfully, but mathematically. The core insight is simple: a computation unit that receives multiple weighted inputs, sums them, and applies a threshold function is both easy to build in code and, when combined in large numbers, astonishingly powerful.
Each input is multiplied by a weight (how important that signal is). A bias shifts the result. An activation function then transforms the sum — introducing the non-linearity that makes deep networks capable of learning complex patterns. The weights are the parameters the network learns during training.
2. A Single Artificial Neuron
Let's make this concrete. A single neuron has these components:
- 1Inputs: x₁, x₂, ..., xₙ — the values fed into this neuron. For an image, these might be pixel brightness values between 0 and 1.
- 2Weights: w₁, w₂, ..., wₙ — one per input. Positive weight = the input helps activate the neuron. Negative weight = the input suppresses it. These are learned.
- 3Bias: b — a single learnable scalar added to the weighted sum. It shifts the activation threshold. Without bias, the neuron can only activate if the weighted inputs are positive.
- 4Weighted sum: z = w₁x₁ + w₂x₂ + ... + wₙxₙ + b — the "pre-activation" value.
- 5Activation function: f(z) — transforms z into the neuron's output. This is what prevents the whole network from collapsing into a single linear equation.
Worked Example: A Brightness-Detection Neuron
Suppose we have a neuron that looks at 3 pixels of an image to decide if a region is "bright." The neuron has been trained and its current weights are:
If all three pixels had been dark (e.g., 0.1, 0.1, 0.1), the weighted sum would be about −0.21, and ReLU would output 0 — "not a bright region." The neuron has learned a simple but meaningful detector.
3. Layers — From Pixels to Predictions
A single neuron can only learn a single simple function. The power of neural networks comes from stacking many neurons into layers, and stacking many layers into a network. Each layer learns to represent the data at a different level of abstraction.
No computation — just holds the data. 1,024 units for a 32×32 image.
Learns edges, brightness gradients, simple textures.
Combines edges into corners, stroke junctions, curves.
10 neurons — one per digit class (0–9). Softmax applied.
Why does depth help? Because each layer builds on what the previous layer learned. If Layer 1 can detect edges, Layer 2 doesn't need to relearn what an edge is — it can combine edge detectors to find corners. If Layer 2 can find corners and curves, Layer 3 can combine them into loops and strokes. By Layer 4, the network has built up enough abstract vocabulary to recognise an entire digit.
The input layer just holds the raw data — no transformation is applied. The hidden layers are where all the learned computation happens. The output layer produces the final prediction — for classification, it typically has one neuron per class, with a softmax function converting raw scores to probabilities.
4. Weights Are Everything
When a network is first created, all its weights are initialised randomly — small random values, often drawn from a normal distribution. At this point, the network's predictions are essentially random too. Training is the process of systematically adjusting those weights until the predictions become accurate.
The total number of weights and biases in a network is called its parameter count. To calculate it for a dense (fully-connected) layer:
Let's count for a concrete example — a network with a 1,024-unit input (flattened 32×32 image), a hidden layer of 64 neurons, and an output layer of 10 neurons:
| Layer | Shape | Weights | Biases | Total |
|---|---|---|---|---|
| Dense 1 | 1024 → 64 | 65,536 | 64 | 65,600 |
| Dense 2 (output) | 64 → 10 | 640 | 10 | 650 |
| Total Parameters | 66,250 | |||
That single hidden layer of 64 neurons already requires 65,600 numbers to be learned. In the SVHN project, ANN Model 1 had 68,010 parameters total — the network was 3 layers deep with 64 hidden units in its single hidden layer. Every one of those 68,010 numbers was initialised randomly and adjusted through training.
5. The Forward Pass
When you feed an input to a neural network and get a prediction out, you are performing a forward pass. Data flows left to right through the network: input layer → hidden layers → output layer. Each layer receives a tensor of values, multiplies by its weight matrix, adds biases, and applies an activation function.
Worked Example: Predicting the Digit "7"
An image of the digit "7" enters the network. After passing through all the layers, the output layer (before softmax) produces 10 raw scores called logits. Softmax converts them to probabilities:
The network is most confident about class 7 (88%), with small residual probabilities spread across the other classes — particularly class 9 (1%) and class 2 (3%), which may have some visual similarity. The argmax of the probability vector gives the final prediction.
6. What ANNs Can't See — The Limits of Flattening
Here is the fundamental problem with applying a standard (fully-connected) neural network to images. When you flatten a 32×32 image into a 1,024-element vector, you destroy the spatial structure of the image.
For a standard neural network, the weight connecting input 163 to a hidden neuron has no relationship to the weight connecting input 195 to the same neuron. The network must learn, independently for every position, that pixels in vertical proximity are likely to be part of the same stroke.
This is enormously inefficient and explains why ANNs plateau quickly on image tasks. Convolutional Neural Networks (CNNs) solve this by keeping the 2D structure intact and sharing weights across spatial positions — one edge-detector filter works everywhere in the image, regardless of position.
ANN Model 1 (68,010 parameters, 3 layers) scored 63.57% test accuracy — barely better than chance for a 10-class problem. ANN Model 2 (310,570 parameters, 5 layers + dropout + batch normalisation) reached 77.33% — a significant gain from regularisation and capacity, but still held back by the fundamental limitation of flattening images. The breakthrough came from switching to CNNs: CNN Model 2 hit 92.22% with fewer parameters than ANN Model 2, simply by preserving spatial structure through convolutional layers.
Interactive: Forward Pass & Backpropagation
Run Forward Pass to make a prediction. Then run Backprop to watch the error signal travel backwards, computing a gradient for every single weight. Or hit Full Training Step to see both in one go — that is exactly one iteration of training.
How Does the Model Know Which Weight to Change?
The gradient ∂L/∂wᵢ tells the model exactly which direction and how much to adjust each weight. Two key rules: the sign tells direction (increase or decrease), and the magnitude tells step size — and it scales with the input value xᵢ.