Hypothesis Testing

The Question Hypothesis Testing Answers

You observe that weekend delivery times average 28 minutes while weekday times average 22 minutes. Is that a real difference — or could it just be random variation in your sample? If you drew a different sample, might the weekday times be higher by chance?

Hypothesis testing is a formal procedure for answering: "Is the pattern I see in my data likely to be real, or likely to be noise?"

It doesn't prove anything. It quantifies how surprising your data would be if the effect you're looking for didn't actually exist.

The Null and Alternative Hypotheses

Every hypothesis test starts with two competing statements:

The logic is deliberately conservative: you never directly "prove" H₁. Instead, you try to show that H₀ is so unlikely given your data that you're willing to reject it in favour of H₁.

The p-Value

The p-value is the probability of observing data as extreme as yours (or more extreme) if the null hypothesis were actually true.

A small p-value means: "If there really were no difference, seeing this data would be very unlikely — so maybe the null is wrong." A large p-value means: "Even if there's no real difference, you'd often see results like this just by chance."

The Significance Threshold (α)

Before running the test, you pick a threshold α (alpha) — typically 0.05. If p < α, you "reject the null hypothesis." The choice of 0.05 is a convention (Ronald Fisher suggested it in the 1920s), not a law of nature. For medical research, α = 0.01 is common. For exploratory data science, 0.05 or 0.10 is standard.

The Welch t-Test

When comparing the means of two groups, the standard tool is the t-test. The Welch t-test is a variant that doesn't assume the two groups have equal variance — making it more robust and the default choice for most real-world comparisons.

The numerator is the observed difference in means. The denominator is a measure of how much sampling variability you'd expect. A large t-statistic means the difference is large relative to the noise — unusual if H₀ were true.

When to use a t-test

• One-sample t-test: Is this sample's mean different from a known value? ("Is our mean delivery time different from the industry benchmark of 30 minutes?")
• Two-sample t-test (Welch): Are the means of two independent groups different? ("Do weekday and weekend orders differ in delivery time?")
• Paired t-test: Are two related measurements different? ("Does the same customer rate orders differently before vs. after a service change?")

Statistical vs. Practical Significance

This is the second most important lesson in hypothesis testing (after understanding the p-value correctly).

Effect size measures translate statistical results into practical terms:

• Cohen's d: Difference in means divided by pooled standard deviation. d = 0.2 (small), 0.5 (medium), 0.8 (large).
• Raw difference with confidence interval: "Weekends are 6.1 minutes slower [CI: 4.8 to 7.4 min]" is more actionable than "p = 0.00001."

Type I and Type II Errors

These errors trade off: lowering α (being stricter) reduces Type I errors but increases Type II errors. The right balance depends on the cost of each type of mistake in your specific context.