Findings from Empirical Within Study Comparisons about the Role of Pretests and Proxy Pretests in Adjusting for Selection Bias in STEM Quasi-Experiments

教育实验设计包括治疗组和对照组，随机分配到不同的干预条件，被认为是能够评估因果关系的最严格的选择。然而，随机对照试验（RCT）在为许多STEM实践、项目和政策的影响提供证据方面往往是不可行的。西北大学的研究人员研究了准实验设计，包括治疗组、对照组和结果测量，但不包括随机分配到治疗和控制条件，如何产生与实验设计相同质量的发现。研究人员还在研究研究结果的预处理措施是否可以用代理预测试取代，代理预测试与原始预测试在同一领域，但形式不同，以使用原始预测试复制实验研究的结果。许多STEM研究经常在准实验研究中访问这些档案数据集中的代理预测试数据，例如州纵向数据系统。这个项目的研究人员已经确定了一些随机对照试验，他们用这些随机对照试验来检验，改变对照组的性质，从最初的随机分配组，到一个从更大的人群中进行统计调整的对照组，是否会显示出与最初的随机对照试验相似的因果估计。通过研究内比较，研究人员确定了随机对照试验和准实验结果的差异。他们正在研究代理预处理测试的使用，例如来自各州纵向数据系统的数学成绩数据，以检查这些措施在多大程度上重复了原始随机对照试验的发现。他们还研究了用一些干预前协变量建模对因果估计的影响。如果RCT和准实验研究的因果估计之间的差异很低，那么准实验设计足以衡量影响。随着地区和地方数据系统的发展，越来越多的大型数据集可用于教育，这些数据系统扩展了所收集的有关学生、教师和学校的信息。这些数据集为研究和评价研究提供了机会，这些研究可以使用人口水平的数据，而不是从人口中随机抽取的数据样本。本研究确定的准实验评估设计的质量为政策制定者和研究人员研究难以研究的教育治疗、项目和政策问题提供了信息。消除随机分配的潜在障碍，同时仍然保持随机对照试验的质量，扩展了评估人员的方法工具包。

Experimental designs in education that include treatment and control groups that are randomly assigned to different conditions of an intervention are considered the most rigorous choice to be able to evaluate causal claims. However, a randomized controlled trial (RCT) is frequently not feasible in developing evidence around the impact of many STEM practices, programs and policies. Researchers from Northwestern University examine how quasi-experimental designs that include treatment and comparison groups and outcome measures, but that do not include the random assignment to treatment and control conditions, might produce the same quality of findings as experimental designs. The researchers are also examining if pretreatment measures of study outcomes might be replaced with proxy pretests, measures in the same domain as an original pretest but in a different form, to replicate the findings from experimental studies using the original pretest. Many STEM research studies frequently have access in quasi-experimental studies for data from these proxy pretests in archival datasets, such as the state longitudinal data systems.Researchers in this project have identified a number of RCTs that they use to examine whether changing the nature of the comparison group, from the original randomly assigned group, to one that is statistically adjusted from a larger population will show similar causal estimates as the original RCT. Using a within-study comparison, the researchers determine the differences in the RCT and quasi-experimental findings. They are studying the use of proxy pretreatment tests, such as math achievement data from state longitudinal data systems, to examine the extent to which these measures replicate findings in the original RCTs. They also examine the effect that modeling with a number of pre-intervention covariates has on causal estimates. If the differences between the causal estimates in the RCT and the new quasi-experimental study are low, then the quasi-experimental design is adequate to measure impact. Large data sets are increasingly available in education, especially with the development of district and local data systems that expand the information gathered about students, teachers and schools. These data sets provide opportunities for research and evaluation studies that can operate with population level data rather than samples of data drawn randomly from the population. The determination of the quality of quasi-experimental evaluation designs that result from this study provides information that can be used by policy makers and researchers to study questions about educational treatments, programs and policies that have been intractable to study. Removing the potential barrier of random assignment, while still maintaining the quality of an RCT, expands the methodological toolkit of evaluators.