Developing a web-tool for increasing the generalizability of large-scale educational evaluations

A central problem in educational evaluation is how to design a study so that the results are both causal and generalizable. The statistical ideal would involve two randomization processes first, schools would be randomly selected from a well-defined inference population, and second, participants would be randomly assigned to treatment conditions. The fact that it is difficult to get schools to agree to take part in randomized experiments makes this dual-randomization process exceedingly rare in practice.Dr. Tipton’s research focuses on the development of statistical tools to improve site selection and recruitment efforts, thus improving causal generalization. In this project, Drs. Tipton and Hedges are developing a webtool to help researchers design targeted recruitment plans and assess generalizability. The tool will guide researchers through first defining the inference population most relevant to the study; second, dividing this population into strata; and third, providing researchers with a list of schools and recruitment goals within each stratum. The use of strata increases the compositional similarity between the sample and population, and helps researchers describe and explain variation in treatment impacts. When completed, the tool will be publically accessible.,