MDM Reference  •  Foundations of inference

Foundations of Statistical Inference

Inference is the movement from what has been observed to what may be said about what has not. The foundations of the subject fix the meaning of that movement, name the quantities it needs, and separate what follows from assumption from what follows from data alone.

Probability as a measure

A probability is a number assigned to an event by a rule that is additive over disjoint events and equal to one over the whole space of outcomes. From this spare beginning the whole structure is built: conditional probability repairs the space when information arrives, and independence names the case in which that repair changes nothing. The reader should hold firmly to the point that probability is a property of a model, not of the world, and that every inference is therefore an inference within a stated model.

The population and the sample

A population is the collection of all units about which a statement is wanted; a sample is the finite part of it that is actually observed. The gap between the two is the source of sampling variation, and the whole of inference is a disciplined response to that gap. An estimator is a rule that turns a sample into a number; its sampling behaviour is described by the distribution it would take over imagined repetitions of the same sampling act.

Likelihood

Given a model indexed by a parameter, the likelihood is the density of the observed sample regarded as a function of that parameter. It is not a probability over parameters, and confusing the two is the commonest error of the elementary reader. The likelihood becomes a probability over parameters only when it is joined to a prior by the rule of Bayes, and that joining is itself a modelling choice rather than a theorem.

Bias, variance and the trade

An estimator is unbiased when its expectation equals the quantity it aims at. Unbiasedness alone is a thin virtue, since a wildly dispersed unbiased rule is worse in practice than a slightly shifted rule of small dispersion. The mean squared error, the sum of the squared bias and the variance, is the quantity that makes the trade visible, and much of estimation theory is a study of how that sum may be reduced under stated restrictions.

Consistency and the large sample

An estimator is consistent when it tends toward the true value as the sample grows without bound. Consistency is a property of a sequence of rules rather than of a single rule, and it is silent about the sample sizes actually met in practice. It remains the least that is asked of an estimator, because a rule that is not consistent cannot be repaired by more data.

The role of assumptions

Every result in this cell rests on assumptions about the model, the sampling act, and the finiteness of the moments involved. When an assumption fails, the result does not become false in general; it simply ceases to hold, and the honest course is to name the failure and to reach for a result whose conditions hold. This habit of naming conditions, kept throughout the reference, is the practical face of sound inference.