Activation Functions

The Problem with Pure Linear Networks

A neural network layer computes: output = weights × input + bias. This is a linear transformation. If you stack 10 such layers, the result is still a linear transformation — you can always collapse 10 linear layers into one equivalent layer. No amount of stacking helps.

Real-world relationships aren't linear. An image of a "5" doesn't differ from an image of a "6" in a way that any straight line can separate. Spoken language, faces, handwriting — all are deeply non-linear in pixel space.

The solution: non-linear activation functions After each layer's linear computation, we pass the result through an activation function — a mathematical transformation that introduces non-linearity. Now stacking layers creates something fundamentally more powerful than a single layer. The network can learn curves, edges, complex shapes, and abstract concepts.

The Core Activation Functions

Hidden layers (default choice)

ReLU — Rectified Linear Unit

f(x) = max(0, x)

If the input is positive, pass it through unchanged. If it's negative, output zero. That's it. Deceptively simple — yet the most commonly used activation function in deep learning for over a decade.

Why it works: It's computationally cheap (one comparison), doesn't suffer from the vanishing gradient problem in the positive range, and is biologically motivated (neurons either fire or they don't).

The dying ReLU problem: If a neuron's input is always negative, it always outputs zero, and its gradient is always zero — the neuron never learns. It's "dead." This happens more often with high learning rates or unlucky weight initialisation. LeakyReLU fixes this.

Hidden layers (when dying ReLU is a concern)

LeakyReLU

f(x) = x if x > 0, else αx (typically α = 0.01)

Like ReLU, but instead of outputting zero for negative inputs, it outputs a small fraction (e.g., 1% of the input). The neuron is still mostly "quiet" for negative inputs, but its gradient is never exactly zero — so it can always recover and keep learning.

When to use: As a drop-in replacement for ReLU when you observe dead neurons or training instability. The SVHN CNN models used LeakyReLU throughout the hidden layers for exactly this reason.

Output layer — multi-class classification

Softmax

f(xᵢ) = e^xᵢ / Σ e^xⱼ (sum over all output classes)

Takes a vector of raw scores (logits) and converts them to a probability distribution that sums to 1.0. For a 10-class digit classifier, it outputs 10 numbers that each represent the model's confidence for that digit, all summing to 100%.

Why it's used at the output: It makes the output interpretable as probabilities. The model's prediction is the class with the highest probability. Paired with cross-entropy loss, it gives clean gradients during training.

Output layer — binary classification

Sigmoid

f(x) = 1 / (1 + e^−x)

Squashes any input to a range between 0 and 1. Useful for binary classification (yes/no, spam/not-spam) where the output represents the probability of the positive class. Not used in the SVHN project (10 classes, not 2), but fundamental to understand.

Vanishing gradient problem: For very large or very small inputs, the sigmoid is nearly flat — gradient ≈ 0. This makes learning very slow in deep networks. This is why ReLU replaced sigmoid as the default hidden-layer activation.

The Vanishing Gradient Problem

During training, the network learns by computing gradients (rates of change) and passing them backwards through the layers (backpropagation). If an activation function "squashes" its input into a very small range, its gradient is also very small.

When you multiply many small gradients together (one per layer), the product becomes astronomically small — effectively zero. Early layers receive almost no learning signal. They stop updating. The network fails to learn.

Why ReLU solves this (partially) For positive inputs, ReLU has a gradient of exactly 1. No squashing, no shrinkage. Gradients pass through unchanged for active neurons. This is why deep networks became practical once ReLU replaced sigmoid and tanh as the default hidden-layer activation — going from 3-4 usable layers to tens or hundreds.

Choosing the Right Activation Function

Hidden layers: Start with ReLU. Switch to LeakyReLU if you observe dying neurons or training stalls.
Output layer (multi-class): Always Softmax.
Output layer (binary): Sigmoid.
Output layer (regression): No activation (or linear) — you want unconstrained output.

Don't use Sigmoid or Tanh in hidden layers Both suffer from vanishing gradients in deep networks. Unless you have a specific architectural reason (e.g., LSTMs use tanh internally), use ReLU or LeakyReLU for hidden layers.

Interactive: Activation Function Plotter

Toggle each function on/off and drag the slider to see the exact output f(x) and its gradient at any input value. The gradient is what flows backwards during training — notice how Sigmoid's gradient collapses near the edges.

f(x) vs x · range −4 to +4

x = 1.0

Drag slider · toggle buttons to compare functions

In the SVHN Project

All hidden layers in both the ANN and CNN models used LeakyReLU (α = 0.1). The output layer used Softmax to produce class probabilities across 10 digit categories (0–9). The switch from ReLU to LeakyReLU was deliberate: with deep architectures and batch normalisation, some neurons were at risk of dying — LeakyReLU prevented this and contributed to stable training. See Chapter 3 — Model Architecture →

← Neural Networks Next: How Networks Learn →