Findings from Empirical Within Study Comparisons about the Role of Pretests and Proxy Pretests in Adjusting for Selection Bias in STEM Quasi-Experiments

教育中的实验设计，包括随机分配到不同干预条件的治疗组和对照组，被认为是能够评估因果关系的最严格选择。然而，随机对照试验（RCT）在围绕许多STEM实践，计划和政策的影响制定证据方面往往不可行。来自西北大学的研究人员研究了包括治疗组和对照组以及结果测量的准实验设计，但不包括随机分配给治疗和对照条件，可能会产生与实验设计相同的结果质量。研究人员还在研究是否可以用替代预测试来取代预测试的研究结果，替代预测试是与原始预测试相同但形式不同的测试，以复制使用原始预测试的实验研究结果。许多STEM研究经常在准实验研究中访问档案数据集中这些代理预测试的数据，例如州纵向数据系统。该项目的研究人员已经确定了一些RCT，他们使用这些RCT来检查是否改变对照组的性质，从最初的随机分配组，从一个更大的人群进行统计学调整后，将显示与原始RCT相似的因果估计。通过研究内比较，研究人员确定了RCT和准实验结果的差异。他们正在研究使用代理预处理测试，例如来自州纵向数据系统的数学成绩数据，以检查这些措施在多大程度上复制了原始RCT中的结果。他们还研究了使用一些干预前协变量建模对因果估计的影响。如果随机对照试验和新的准实验研究中的因果估计之间的差异很小，那么准实验设计足以衡量影响。 在教育方面，大型数据集越来越多，特别是随着地区和地方数据系统的发展，扩大了收集的关于学生、教师和学校的信息。这些数据集为研究和评价研究提供了机会，这些研究和评价研究可以使用人口一级的数据，而不是从人口中随机抽取的数据样本。从这项研究中产生的准实验性评估设计的质量的确定提供了可供决策者和研究人员使用的信息，以研究有关教育治疗，计划和政策的问题，一直难以研究。消除随机分配的潜在障碍，同时保持RCT的质量，扩大了评价者的方法工具包。

Experimental designs in education that include treatment and control groups that are randomly assigned to different conditions of an intervention are considered the most rigorous choice to be able to evaluate causal claims. However, a randomized controlled trial (RCT) is frequently not feasible in developing evidence around the impact of many STEM practices, programs and policies. Researchers from Northwestern University examine how quasi-experimental designs that include treatment and comparison groups and outcome measures, but that do not include the random assignment to treatment and control conditions, might produce the same quality of findings as experimental designs. The researchers are also examining if pretreatment measures of study outcomes might be replaced with proxy pretests, measures in the same domain as an original pretest but in a different form, to replicate the findings from experimental studies using the original pretest. Many STEM research studies frequently have access in quasi-experimental studies for data from these proxy pretests in archival datasets, such as the state longitudinal data systems.Researchers in this project have identified a number of RCTs that they use to examine whether changing the nature of the comparison group, from the original randomly assigned group, to one that is statistically adjusted from a larger population will show similar causal estimates as the original RCT. Using a within-study comparison, the researchers determine the differences in the RCT and quasi-experimental findings. They are studying the use of proxy pretreatment tests, such as math achievement data from state longitudinal data systems, to examine the extent to which these measures replicate findings in the original RCTs. They also examine the effect that modeling with a number of pre-intervention covariates has on causal estimates. If the differences between the causal estimates in the RCT and the new quasi-experimental study are low, then the quasi-experimental design is adequate to measure impact. Large data sets are increasingly available in education, especially with the development of district and local data systems that expand the information gathered about students, teachers and schools. These data sets provide opportunities for research and evaluation studies that can operate with population level data rather than samples of data drawn randomly from the population. The determination of the quality of quasi-experimental evaluation designs that result from this study provides information that can be used by policy makers and researchers to study questions about educational treatments, programs and policies that have been intractable to study. Removing the potential barrier of random assignment, while still maintaining the quality of an RCT, expands the methodological toolkit of evaluators.