A novel method to obtain the treatment effect assessed for a completely randomized design: Multiple imputation of unobserved potential outcomes

Abstract

In this paper a novel method to obtain the treatment effect called multiple imputation of unobserved potential outcomes is evaluated by comparing its performance to the Student’s t-test and ANCOVA. The novel method originates from Rubins potential outcomes framework which explicitly defines that each unit has a possible active treatment and control treatment outcome. Since every unit can only be assigned to one treatment, one of the potential outcomes is unobserved for each unit. To approximate calculating the treatment effect, the unobserved potential outcomes are multiple imputed. To start at the basics of group comparisons, the paper is restricted to completely random groups. The method is evaluated by a series of simulations using a realistic, empirical synthetic population. Overall, the results show that the novel method performs up to standards: bias is negligible compared to the standard error and 95 per cent confidence interval coverage is above 90 per cent. Also, the novel method is more efficient and powerful than the frequently used Student’s t-test when the relation between the covariates and potential outcomes is linear. Multiple imputation of potential outcomes performs approximatly equally well as classical ANCOVA. At a small sample size, the novel method is somewhat more powerful when the assumption of parallel slopes is violated, but it is slightly less efficient than ANCOVA for all used properties of covariates, sample sizes and effect sizes.