Elizabeth Tipton: Designing Randomized Trials to Predict Treatment Effects

Speaker: Elizabeth Tipton (Northwestern University)
Discussant: Andrew Gelman (Columbia University)
Title: Designing Randomized Trials to Predict Treatment Effects
Abstract: Typically, a randomized experiment is designed to test a hypothesis about the average treatment effect and sometimes hypotheses about treatment effect variation. The results of such a study may then be used to inform policy and practice for units not in the study. In this paper, we argue that given this use, randomized experiments should instead be designed to predict unit-specific treatment effects in a well-defined population. We then consider how different sampling processes and models affect the bias, variance, and mean squared prediction error of these predictions. To do so, we derive formulas - similar to those in a power analysis - based upon parametric models. The results indicate, for example, that problems of generalizability — differences between samples and populations — can greatly affect bias both in predictive models and in measures of error in these models. We also examine when the average treatment effect estimate outperforms unit-specific treatment effect predictive models and implications of this for planning studies.