MDM Reference  •  Data analysis workflow

The Data Analysis Workflow

Analysis proceeds best in stages, each with a clear question and a clear check. The sequence below is not a rigid ritual, but it names the steps that keep a study honest and its conclusions traceable.

State the question first

Before any number is computed, the analyst should be able to say what quantity is being estimated and why it matters. A question phrased in terms of a parameter and a population is a question that can be answered; a question phrased in terms of a desired conclusion is not. The first act of discipline is to fix the target before looking at the data.

Inspect before modelling

A first pass should establish the shape of each variable, the presence of missing readings, and the pattern of extreme values. A histogram, a step curve of the ordered values, or a scatter of one variable against another will usually reveal more than a summary table. Anomalies found here will govern every later choice.

Choose a model and state its assumptions

The model is the bridge between the population and the sample. Its assumptions, whether of independence, of a stated family, or of constant dispersion, must be written down, because they are the conditions under which the later statements hold. A model chosen for convenience must be acknowledged as such.

Fit and estimate

Fitting turns the sample into estimated parameters. Whatever method is used, the estimates should be reported with their uncertainty, not as bare numbers. The analyst should be able to say how the estimates would vary under a repetition of the study, and an interval or a standard error is the usual way of saying it.

Check the fit

Checks ask whether the data contradict the model in any way the model cares about. Residuals, the differences between observed and fitted values, should be inspected for pattern; a systematic shape in the residuals points to a missing feature of the model. A check that is skipped is a claim that is unsupported.

Report and interpret

The closing stage states what was found, how uncertain it is, and what it does not establish. Conventions of language matter here: an association is not a cause, a sample is not a population, and a model is not a fact. A report that respects these distinctions is one that a reader may rely upon, and it is the standard this compendium tries to hold.

Keeping a record of the analysis

A study that is not recorded cannot be checked, and a study that cannot be checked is of little weight. The analyst should keep, beside the estimates themselves, the choices that produced them: the readings set aside, the variables transformed, the model fitted, and the order in which these steps were taken. The record need not be long, but it should be complete enough for another worker to retrace the path.

When the question changes

It often occurs that the first look at the data suggests a question other than the one first put. The honest course is to name the change openly rather than to present the later question as though it had been the original. A question settled after a look at the data is still worth answering; it is only the claim of having known it all along that misleads.