12.2 Reporting

Section 1.1.2 noted that the attitude toward missing data is changing. Many aspects related to missing data could potentially affect the conclusions drawn for the statistical analysis, but not all aspects are equally important. This leads to the question: What should be reported from an analysis with missing data?

Guidelines to report the results of a missing data analysis have been given by Sterne et al. (2009), Enders (2010), National Research Council (2010) and Mackinnon (2010). These sources vary in scope and comprehensiveness, but they also exhibit a great deal of overlap and consensus. Section 12.2.1 combines some of the material found in the three sources.

Reviewers or editors may be unfamiliar with, or suspicious of, newer approaches to handling missing data. Substantive researchers are therefore often wary about using advanced statistical methods in their reports. Though this concern is understandable,

\(\dots\) resorting to flawed procedures in order to avoid criticism from an uninformed reviewer or editor is a poor reason for avoiding sophisticated missing data methodology (Enders 2010, 340)

Until reviewers and referees become more familiar with the newer methods, a better approach is to add well-chosen and concise explanatory notes. On the other hand, editors and reviewers are increasingly expecting applied researchers to do multiple imputation, even when the authors had good reasons for not doing it (e.g., less than 5% incomplete cases) (Ian White, personal communication).

The natural place to report about the missing data in a manuscript is the paragraph on the statistical methodology. As scientific articles are often subject to severe space constraints, part of the report may need to go into supplementary online materials instead of the main text. Since the addition of explanatory notes increases the number of words, there needs to be some balance between the material that goes into the main text and the supplementary material. In applications that requires novel methods, a separate paper may need to be written by the team’s statistician. For example, Van Buuren, Boshuizen, and Knook (1999) explained the imputation methodology used in the substantive paper by Boshuizen et al. (1998). In general, the severity of the missing data problem and the method used to deal with the problem needs to be part of the main paper, whereas the precise modeling details could be relegated to the appendix or to a separate methodological paper.

12.2.1 Reporting guidelines

The following list contains questions that need to be answered when using multiple imputation. Evaluate each question carefully, and report the answers.

1. *Amount of missing data:* What is the number of missing values for each variable of interest? What is the number of cases with complete data for the analyses of interest? If people drop out at various time points, break down the number of participants per occasion.

2. Reasons for missingness: What is known about the reasons for missing data? Are the missing data intentional? Are the reasons possibly related to the outcome measurements? Are the reasons related to other variables in the study?

3. Consequences: Are there important differences between individuals with complete and incomplete data? Do these groups differ in mean or spread on the key variables? What are the consequences if complete-case analysis is used?

4. Method: What method is used to account for missing data (e.g., complete-case analysis, multiple imputation)? Which assumptions were made (e.g., missing at random)? How were multivariate missing data handled?

5. Software: What multiple imputation software is used? Which settings differ from the default?

6. Number of imputed datasets: How many imputed datasets were created and analyzed?

7. Imputation model: Which variables were included in the imputation model? Was any form of automatic variable predictor used? How were non-normally distributed and categorical variables imputed? How were design features (e.g., hierarchical data, complex samples, sampling weights) taken into account?

8. Derived variables: How were derived variables (transformations, recodes, indices, interaction terms, and so on) taken into account?

9. Diagnostics: How has convergence been monitored? How do the observed and imputed data compare? Are imputations plausible in the sense that they could have been plausibly measured if they had not been missing?

10. Pooling: How have the repeated estimates been combined (pooled) into the final estimates? Have any statistics been transformed for pooling?

11. Complete-case analysis: Do multiple imputation and complete-case analysis lead to similar similar conclusions? If not, what might explain the difference?

12. Sensitivity analysis: Do the variables included in the imputation model make the missing at random assumption plausible? Are the conclusions affected if imputations are generated under a plausible nonignorable model?

If space is limited, the main text can be restricted to a short summary of points 1, 2, 4, 5, 6 and 11, whereas the remaining points are addressed in an appendix or online supplement. Section 12.2.2 contains an example template.

For clinical trials, reporting in the main text should be extended by point 12, conform to recommendation 15 of National Research Council (2010). Moreover, the study protocol should specify the statistical methods for handling missing data in advance, and their associated assumptions should be stated in a way that can be understood by clinicians (National Research Council 2010 recommendation 9).

12.2.2 Template

Enders (2010, 340–43) provides four useful templates for reporting the results of a missing data analysis. These templates include explanatory notes for uninformed editors and reviewers. It is straightforward to adapt the template text to other settings. Below I provide a template loosely styled after Enders that I believe captures the essentials needed to report multiple imputation in the statistical paragraph of the main text.

The percentage of missing values across the nine variables varied between 0 and 34%. In total 1601 out of 3801 records (42%) were incomplete. Many girls had no score because the nurse felt that the measurement was “unnecessary,” or because the girl did not give permission. Older girls had many more missing data. We used multiple imputation to create and analyze 40 multiply imputed datasets. Methodologists currently regard multiple imputation as a state-of-the-art technique because it improves accuracy and statistical power relative to other missing data techniques. Incomplete variables were imputed under fully conditional specification, using the default settings of the mice 3.0 package (Van Buuren and Groothuis-Oudshoorn 2011). The parameters of substantive interest were estimated in each imputed dataset separately, and combined using Rubin’s rules. For comparison, we also performed the analysis on the subset of complete cases.

This text is about 135 words. If this is too long, then the sentences that begin with “Methodologists” and “For comparison” can be deleted. In the paragraphs that describe the results we can add the following sentence:

Table 1 gives the missing data rates of each variable.

In addition, if complete-case analysis is included, then we need to summarize it. For example:

We obtained similar results when the analysis was restricted to the complete cases only. Multiple imputation was generally more efficient as can be seen from the shorter confidence intervals and lower \(p\)-values in Table X.

It is also possible that the two analyses lead to diametrically opposed conclusions. Since a well-executed multiple imputation is theoretically superior to complete-case analysis, we should give multiple imputation more weight. It would be comforting though to have an explanation of the discrepancy.

The template texts can be adapted as needed. In addition obtain inspiration from good articles in your own field that apply multiple imputation.