MDM Reference  •  Covariance & dispersion

Covariance, Dispersion and Dependence

Dispersion is the width of a distribution, and covariance is the joint width of a pair. Together they measure how much a set of observations wanders, and how much of that wandering is shared.

Variance as a squared distance

The variance of a variable is the expectation of the squared departure from its mean. It is a squared distance, so its units are squared, and it is for that reason that its square root, the standard deviation, is quoted when a width on the original scale is wanted. The variance is never negative and is zero only for a variable that takes a single value with certainty.

Addition rules

The variance of a sum is the sum of the variances plus twice the covariance. This one rule carries most of the practical arithmetic of dispersion. When the terms are independent the covariance vanishes and the variances simply add; when the terms move together the covariance swells the total; and when they move oppositely it reduces the total, which is the arithmetic behind the cancelling of errors.

Covariance and its limits

Covariance measures joint linear movement, but its size depends on the units of the two variables, so it cannot be compared across pairs without standardisation. Dividing by the two standard deviations yields the correlation, a pure number confined between minus one and one. The endpoints are reached only under exact linear relation, and the middle is reached both by genuine independence and by certain symmetric non-linear relations.

The covariance arrangement as geometry

The square arrangement of covariances describes an ellipsoid of dispersion whose axes point along its eigenvectors and whose lengths are the square roots of its eigenvalues. This is the geometric content of the multivariate theory, and it explains why the arrangement must be positive semidefinite: a negative eigenvalue would describe a direction of negative spread, which is impossible.

Partitioned dispersion

When a set of variables is split into two blocks, the arrangement splits with it, and the conditional dispersion of one block given the other is the difference between the marginal arrangement and a term built from the cross block. This is the algebraic core of regression, and it shows that conditioning can only reduce dispersion, never increase it.

Working remarks

Dispersion is not the same as range, and a small range in a sample need not imply a small variance in the population. Outliers inflate the variance sharply, since their departures are squared, and a single extreme reading can dominate the total. The reader is reminded to inspect the shape of a distribution before quoting a single measure of width alongside it.

A worked illustration

Consider two quantities that both rise over a range but at differing rates. Their covariance is large and positive, and their correlation, once each is standardised, is near one, because the two move together in a near-linear way. Add a third quantity that rises and then falls, and its correlation with each of the first two is small, though a curved relation plainly exists. The example shows why a small correlation is not the same as the absence of relation.

Dispersion across groups

When readings fall into groups, dispersion may be examined within the groups and between them. A large dispersion between the group centres alongside a small dispersion within each group points to a real grouping, while the reverse points to groups that differ little in the quantity of interest. The two dispersions together tell a fuller story than either alone.