MDM Reference  •  Sampling & estimation

Sampling and Estimation Theory

Estimation is the art of extracting a single number, or a short interval, from a sample of irregular readings. The theory of estimation describes how such a number behaves when the sampling act is imagined to be repeated.

The sampling distribution

Every statistic computed from a sample is itself a random variable, with its own distribution over imagined repetitions. That distribution, not the single observed value, is what licenses a claim about precision. Its spread is governed by the sample size, narrowing in proportion to the square root of the number of readings, a rate that sets a firm limit on what any increase in sample can gain.

Point estimation

A point estimator is a rule that maps a sample to a proposed value. The sample mean estimates the population mean, the sample variance estimates the population variance, and the sample proportion estimates a population share. Each rule carries its own bias and its own dispersion, and the two together fix its mean squared error.

Invariance and sufficiency

Some statistics carry all that a sample can say about a parameter; such a statistic is called sufficient, and no further reduction loses information. Others are ancillary, carrying no information about the parameter at all but useful for checking the model. The search for sufficient statistics is the search for the smallest faithful summary of the data.

Interval estimation

An interval estimate names a range together with a level of confidence. The confidence belongs to the rule, not to the particular interval: over repeated sampling, a stated share of constructed intervals would contain the true value. To say that a single computed interval has a stated probability of containing the value misreads the claim, and the honest wording is careful on this point.

Maximum likelihood

The likelihood method selects the parameter value under which the observed sample is most plausible. Under regularity conditions the resulting estimator is consistent and, in large samples, achieves the smallest dispersion attainable by any regular rule. The qualification about regularity is not decorative: the method fails at boundaries and in models with unidentified parameters.

Resampling as a check

When the sampling distribution is hard to derive, the same reasoning may be rehearsed by drawing many samples from the observed data itself. This rehearsal yields an estimated picture of sampling variation without a formula, and it is a useful cross-check on any closed result. It remains an estimate, and it inherits the limitations of the original sample.

A worked reading of the rate

Consider two studies of one quantity, the first resting on a hundred readings and the second on four hundred. The second has four times the readings and therefore half the width in its estimate of the mean, since width varies with the square root of the count. To halve the width again would call for sixteen hundred readings in all. The arithmetic is plain, and it sets a firm ceiling on what patience with a larger sample may achieve.

Choosing among estimators

When two rules are available for the same quantity, the choice is usually made on the sum of squared bias and dispersion. A rule that is slightly biased but much tighter may serve better than an unbiased rule that wanders widely. The choice should be stated, since a reader who assumes unbiasedness where a biased rule was used will misread the account.