The logic
A hypothesis test asks: if a claim about the population were true, would my sample result be surprising? It follows a courtroom logic: assume innocence (the null) until the evidence is strong enough to doubt it.
Setting up
- Null hypothesis H₀: the status quo or "no effect", with an equals sign. Example: μ = 100.
- Alternative hypothesis Hₐ: what we suspect or want to show: μ ≠ 100 (two-sided), μ > 100 or μ < 100 (one-sided).
- Significance level α: the cut-off for "surprising enough", commonly 0.05. It is the probability of a Type I error we accept.
The test statistic and p-value
The test statistic measures how far the sample result is from the null value, in standard errors. For a mean with known σ:
z = (x̄ − μ₀) / (σ/√n)
The p-value is the probability, assuming H₀ is true, of getting a result at least as extreme as the one observed.
Decision rule:
- If p-value ≤ α: reject H₀ (statistically significant evidence for Hₐ).
- If p-value > α: fail to reject H₀ (not enough evidence). This is not proof that H₀ is true.
Errors
| H₀ true | H₀ false | |
|---|---|---|
| Reject H₀ | Type I error (false alarm) | Correct |
| Fail to reject | Correct | Type II error (missed effect) |
The probability of a Type I error is α. Power = 1 − P(Type II) is the chance of detecting a real effect. Power rises with a bigger sample, a larger true effect and a larger α.
Worked example
A bakery claims its loaves have a mean mass of 100 g. We suspect it differs. A sample of n = 36 loaves gives x̄ = 104 g. Assume σ = 12 g. Test at α = 0.05.
- H₀: μ = 100. Hₐ: μ ≠ 100 (two-sided).
- SE = 12 / √36 = 2.
- z = (104 − 100) / 2 = 2.
- Two-sided p-value = 2 × P(Z > 2) = 2 × 0.0228 ≈ 0.0455.
- 0.0455 < 0.05, so reject H₀.
- Conclusion: there is statistically significant evidence that the mean mass differs from 100 g.
At α = 0.01 the same data would not be significant, since 0.0455 > 0.01. The significance level should be chosen before looking at the data.
Interpreting a p-value
A p-value of 0.04 does not mean there is a 4 percent chance the null is true. It means that if H₀ were true, results this extreme would occur about 4 percent of the time.
Statistical significance is not the same as practical importance: with a huge sample, a trivial difference can be significant.
Common mistakes
- Saying "accept H₀" rather than "fail to reject H₀".
- Misreading the p-value as the probability that H₀ is true.
- Choosing a one-sided test after seeing which way the data point.