Enhancing external validity in existing STEM evaluations

鉴于问责压力越来越大，PI建议开发、描述和测试统计方法，以减少在非代表性站点样本中进行的STEM项目随机分配评估中的外部有效性偏差。我们将利用真实世界的信息进行一系列模拟，以测试不同统计方法在何种条件下可以减少外部效度偏差。方法论方法考察了来自四种研究的真实世界数据：交互回归、贝叶斯加性回归树(BART)、选择加权的逆概率(IPSW)和子分类。在拟议的模拟研究中使用真实世界数据将检验每种方法在减少此类偏差方面的有效性。这项工作将产生四个对STEM评估者和研究人员非常有用的产品：(1)更好地了解参与影响评估的学校和地区的类型；(2)研究人员可以用来提高现有评估的外部有效性的分析方法；(3)关于在未来评估中收集的数据类型的指导；以及研究人员可以用来进行模拟研究以调查其他方法学问题的数据集和框架。

In light of increasing accountability pressures, the PI proposes to develop, describe and test statistical methods for reducing external validity bias in random assignment evaluations of STEM programs carried out in a non-representative samples of sites. A series of simulations will be conducted using real-world information to test the conditions under which different statistical methods can reduce the external validity bias.The methodological approach examines real world data from four kinds of studies: regression with interactions; Bayesian additive regression trees (BART); inverse probablility of selection weighting (IPSW); and Subclassification.The use of real world data in the proposed simulation studies will test the efficacy of each in reducing such bias. Four products will result from the work that will be highly useful to STEM evaluators and researchers: (1) a better understanding of the types of schools and districts that participate in impact evaluation; (2) analysis methods that researchers can use to improve the external validity of existing evaluations; (3) guidance on types of data to collect in future evaluations; and a data set and framework researchers can use to conduct simulation studies to investigate other methodological questions.