Enhancing external validity in existing STEM evaluations

鉴于问责压力的增加，PI建议开发，描述和测试统计方法，以减少在非代表性样本中进行的STEM项目随机分配评估中的外部有效性偏差。 本研究将利用真实世界的数据进行一系列的模拟实验，以检验不同统计方法在何种条件下可以减少外部效度偏差，方法学方法主要考察了来自四种研究的真实的真实世界数据：交互作用回归、贝叶斯加性回归树（BART）、选择加权逆概率（IPSW）、贝叶斯加性回归树（BART）、贝叶斯加性回归树（BART）和贝叶斯加性回归树（IPSW）。在所提出的模拟研究中使用真实的世界数据将测试每一个在减少这种偏差方面的功效。这项工作将产生四个对STEM评估人员和研究人员非常有用的产品：（1）更好地了解参与影响评估的学校和地区的类型;（2）研究人员可以用来提高现有评估的外部有效性的分析方法;（3）关于未来评估中收集的数据类型的指导;（4）关于评估的数据类型的指导。以及一个数据集和框架，研究人员可以用来进行模拟研究，以调查其他方法学问题。

In light of increasing accountability pressures, the PI proposes to develop, describe and test statistical methods for reducing external validity bias in random assignment evaluations of STEM programs carried out in a non-representative samples of sites. A series of simulations will be conducted using real-world information to test the conditions under which different statistical methods can reduce the external validity bias.The methodological approach examines real world data from four kinds of studies: regression with interactions; Bayesian additive regression trees (BART); inverse probablility of selection weighting (IPSW); and Subclassification.The use of real world data in the proposed simulation studies will test the efficacy of each in reducing such bias. Four products will result from the work that will be highly useful to STEM evaluators and researchers: (1) a better understanding of the types of schools and districts that participate in impact evaluation; (2) analysis methods that researchers can use to improve the external validity of existing evaluations; (3) guidance on types of data to collect in future evaluations; and a data set and framework researchers can use to conduct simulation studies to investigate other methodological questions.