Model Evaluation

Why Accuracy is Misleading

A spam filter that labels every single email as "not spam" achieves 99% accuracy if only 1% of emails are actually spam. It's also completely useless. Accuracy hides everything interesting when classes are imbalanced or when different types of mistakes have different costs.

For the SVHN digit recognition task, accuracy is more meaningful — 10 balanced classes, and misclassifying a "3" as an "8" is roughly as bad as any other mistake. But even here, the pattern of errors matters: which digits get confused with which?

The imbalanced class trap Imagine a disease screening test on a population where 2% have the disease. A model that always predicts "healthy" achieves 98% accuracy. Zero useful predictions. Precision and recall exist specifically to expose this failure.

The Confusion Matrix

A confusion matrix is a grid where rows represent the true class and columns represent the predicted class. Each cell shows how many examples of class X were classified as class Y. The diagonal is correct predictions; everything off-diagonal is a mistake.

Simplified 3-class confusion matrix (digits 1, 2, 3):

                Predicted 1   Predicted 2   Predicted 3
True label 1       892            8             4        → 98.7% correct
True label 2        12          860            30        → 94.5% correct
True label 3         5           41           801        → 94.7% correct

Reading column "Predicted 2":
  8 of those were actually 1s  (false positives for class 2)
  860 were actually 2s         (true positives)
  41 were actually 3s          (false positives for class 2)

The matrix shows: digit 3 is frequently confused with 2 (41 cases)
— visually similar curved shapes.

The confusion matrix reveals which specific classes are confused with each other — actionable information that a single accuracy number never could.

The Four Outcomes

For any given class (say, "is this a 5?"), every prediction falls into one of four categories:

True Positive (TP)

The model predicted "5" and it was actually a 5. Correct positive.

True Negative (TN)

The model predicted "not 5" and it wasn't a 5. Correct negative.

False Positive (FP)

The model predicted "5" but it was something else. Wrong — a false alarm. Also called Type I error.

False Negative (FN)

The model predicted "not 5" but it was actually a 5. Missed it. Also called Type II error.

Precision, Recall, and F1

Precision Precision = TP / (TP + FP)

Of all the times the model predicted this class, how often was it right? High precision means few false alarms. "When I say it's a 5, I'm usually correct."

Recall (Sensitivity) Recall = TP / (TP + FN)

Of all the actual examples of this class, how many did the model find? High recall means few misses. "Of all the real 5s, I found most of them."

The precision–recall trade-off These two metrics trade off against each other. A model that only predicts "5" when it's 100% certain will have perfect precision but low recall (it misses most 5s). A model that labels everything "5" will have perfect recall but terrible precision. You tune the threshold based on which error is more costly for your application.

F1-Score — The Harmonic Mean

F1 = 2 · (Precision × Recall) / (Precision + Recall)

The F1-score combines precision and recall into one number. It's the harmonic mean — which penalises large imbalances. A model with precision=1.0 and recall=0.01 gets an F1 of 0.02, not 0.5. This makes F1 a robust single-number summary: you can only get a high F1 if both precision and recall are reasonably high.

Macro vs. Weighted Average

For multi-class problems, you have a precision, recall, and F1 per class. Averaging them:

Macro average: Simple mean across all classes. Treats each class equally regardless of size.
Weighted average: Weights each class by its number of examples. More representative when class sizes differ. Standard for imbalanced datasets.

Reading a Classification Report

CNN2 Classification Report (SVHN test set):

              precision  recall  f1-score  support
           0     0.93    0.91      0.92     1744
           1     0.96    0.97      0.96     4915
           2     0.91    0.90      0.90     4330
           3     0.90    0.88      0.89     2995
           4     0.93    0.93      0.93     2525
           5     0.89    0.88      0.89     2244
           6     0.93    0.93      0.93     1964
           7     0.93    0.94      0.93     2891
           8     0.89    0.91      0.90     2443
           9     0.91    0.92      0.91     2745

    accuracy                       0.922    28796
   macro avg     0.92    0.92      0.92    28796
weighted avg     0.92    0.92      0.92    28796

Observations:
• Digit 1 scores highest (F1=0.96) — distinctive vertical stroke
• Digits 5 and 8 score lowest (F1=0.89) — visually similar curves
• No class is catastrophically worse than others — balanced learning

Train / Validation / Test Split

Evaluating a model on data it trained on is meaningless — it's like grading a student on the same questions used for practice. Proper evaluation requires held-out data:

Training set: The model sees this data and learns from it. Typically 70–80% of the data.
Validation set: Used during training to monitor overfitting and tune hyperparameters. The model doesn't train on this, but it's used to make decisions about training — so it's "seen" indirectly.
Test set: Completely held out. Only used once, at the very end, to report final performance. This is the honest number. If you tune your model based on test set performance, you've contaminated it — it's no longer an unbiased estimate.

The golden rule of evaluation The test set is touched exactly once. Every decision you make during model development — architecture, regularisation, learning rate, number of epochs — is made based on training and validation performance only. The test set tells you how well the model will do on data it has genuinely never influenced.

Interactive: CNN2 Confusion Matrix

Each row is a true digit; each column is what the model predicted. The bright green diagonal is correct predictions. Off-diagonal cells reveal which digit pairs get confused — hover any cell to inspect. Notice how 3↔5 and 4↔9 are the most common errors, driven by visual similarity.

CNN2 · Test Set · 28,796 images · 92.22% accuracy Hover cells to inspect · Rows = true label · Cols = predicted

Overall accuracy: 92.22% — hover any cell to see count and class details

In the SVHN Project

All four models were evaluated on the same held-out test set (28,796 images). The classification report for CNN2 showed consistently high F1 scores across all 10 digits (0.89–0.96), confirming balanced learning with no catastrophically weak class. The confusion matrix revealed that the most common errors were between visually similar digit pairs: 3↔5, 4↔9, and 8↔6. Overall test accuracy: 92.22% — the headline metric, but the full report told the complete story. See Chapter 8 — Model Results →

← Convolutional Networks Course Index →