Enhancing external validity in existing STEM evaluations

鉴于日益增加的问责压力，PI建议开发，描述和测试统计方法，以减少在非代表性站点样本中进行的STEM项目随机分配评估中的外部效度偏差。我们将利用现实世界的信息进行一系列的模拟，以测试不同的统计方法在哪些条件下可以减少外部效度偏差。方法方法检查来自四种研究的真实世界数据：具有相互作用的回归；贝叶斯加性回归树；选择加权逆概率；和子分类。在拟议的模拟研究中使用真实世界的数据将测试每种方法在减少这种偏差方面的功效。这项工作将产生四个对STEM评估人员和研究人员非常有用的产品：(1)更好地了解参与影响评估的学校和地区的类型；(2)研究人员可用于提高现有评价外部效度的分析方法；(3)指导今后评价收集的数据类型；研究人员可以使用数据集和框架进行模拟研究，以调查其他方法论问题。

In light of increasing accountability pressures, the PI proposes to develop, describe and test statistical methods for reducing external validity bias in random assignment evaluations of STEM programs carried out in a non-representative samples of sites. A series of simulations will be conducted using real-world information to test the conditions under which different statistical methods can reduce the external validity bias.The methodological approach examines real world data from four kinds of studies: regression with interactions; Bayesian additive regression trees (BART); inverse probablility of selection weighting (IPSW); and Subclassification.The use of real world data in the proposed simulation studies will test the efficacy of each in reducing such bias. Four products will result from the work that will be highly useful to STEM evaluators and researchers: (1) a better understanding of the types of schools and districts that participate in impact evaluation; (2) analysis methods that researchers can use to improve the external validity of existing evaluations; (3) guidance on types of data to collect in future evaluations; and a data set and framework researchers can use to conduct simulation studies to investigate other methodological questions.