MDM Reference • Hypothesis & significance
Hypothesis Testing and Significance
Hypothesis assessment asks whether the data are compatible with a stated position. It is a formal comparison between what a sample shows and what a proposed model predicts, and its vocabulary is easily abused.
Null and alternative
The null hypothesis states a restriction on the model, often that a parameter takes a particular value or that two quantities do not differ. The alternative states the departures that are of interest. The test is a rule that either rejects the null in favour of the alternative or declines to reject it, and the rule is chosen so that the two kinds of mistake are held at acceptable rates.
The two errors
- A rejection of a true null is an error of the first kind, whose rate is fixed in advance.
- A failure to reject a false null is an error of the second kind, whose complement is the power of the test.
These two rates pull against each other: reducing one, at a fixed sample size, raises the other. Only a larger sample loosens the tension, and this is the reason power deserves as much attention as the rejection rate.
What a p-value is
A p-value is the probability, computed under the null, of a result at least as extreme as the one observed. It is not the probability that the null is true, and it is not the probability that the result arose by chance. A small value says that the observed result would be unusual if the null held; it does not by itself say that the null is false, nor that the departure is large or important.
Significance and importance
With a large sample almost any departure, however small, becomes significant. Significance therefore speaks to detectability, while importance speaks to size. The reader is urged to quote the estimated size of an effect alongside any statement of significance, so that the two are not confused.
Multiplicity
When many hypotheses are assessed at once, the chance that some rejection is spurious rises sharply with the number of tests. The correction of this drift, whether by holding the overall rejection rate or by controlling the expected share of false rejections, is a necessary discipline rather than an optional refinement.
Beyond the threshold
A single threshold is a blunt summary of evidence. The interval estimate, which carries both the size of an effect and the uncertainty attached to it, is usually the more informative report, and modern practice tends to prefer it. The comparison of a value to a threshold remains useful, but it should stand beside the interval rather than in place of it.
A worked reading of power
Consider a test fixed to reject a true null one time in twenty. Its ability to detect a real departure depends on how large that departure is and on how many readings support the comparison. A small departure will often go undetected even when it is real, and the failure to detect it is not evidence that it is absent. The reader should ask, before trusting a failure to reject, whether the study had the means to detect a departure of the size that would matter.
Two tests of one question
Where several tests bear on one question, they rarely agree in full, and the reader is left to weigh them. The desk advises attention to the assumptions each test carries and to the size of the effect each reports, rather than to a count of how many tests reached a threshold. A tally of thresholds can hide disagreement over the very quantity that matters.