Findings from Empirical Within Study Comparisons about the Role of Pretests and Proxy Pretests in Adjusting for Selection Bias in STEM Quasi-Experiments

关于预测试和代理预测试在调整 STEM 准实验中选择偏差中的作用的实证研究比较结果

• 批准号: 1228866
• 负责人: Thomas Cook
• 金额: $ 79.02万
• 依托单位: Northwestern University
• 依托单位国家: 美国
• 项目类别: Standard Grant
• 财政年份: 2012
• 资助国家: 美国
• 起止时间: 2012-10-01 至 2016-09-30
• 项目状态: 已结题
• 来源: https://www.nsf.gov/awardsearch/showAward?AWD_ID=1228866&HistoricalAwards=false
• 关键词: Findings; Empirical; Within; Study; Comparisons

Experimental designs in education that include treatment and control groups that are randomly assigned to different conditions of an intervention are considered the most rigorous choice to be able to evaluate causal claims. However, a randomized controlled trial (RCT) is frequently not feasible in developing evidence around the impact of many STEM practices, programs and policies. Researchers from Northwestern University examine how quasi-experimental designs that include treatment and comparison groups and outcome measures, but that do not include the random assignment to treatment and control conditions, might produce the same quality of findings as experimental designs. The researchers are also examining if pretreatment measures of study outcomes might be replaced with proxy pretests, measures in the same domain as an original pretest but in a different form, to replicate the findings from experimental studies using the original pretest. Many STEM research studies frequently have access in quasi-experimental studies for data from these proxy pretests in archival datasets, such as the state longitudinal data systems.Researchers in this project have identified a number of RCTs that they use to examine whether changing the nature of the comparison group, from the original randomly assigned group, to one that is statistically adjusted from a larger population will show similar causal estimates as the original RCT. Using a within-study comparison, the researchers determine the differences in the RCT and quasi-experimental findings. They are studying the use of proxy pretreatment tests, such as math achievement data from state longitudinal data systems, to examine the extent to which these measures replicate findings in the original RCTs. They also examine the effect that modeling with a number of pre-intervention covariates has on causal estimates. If the differences between the causal estimates in the RCT and the new quasi-experimental study are low, then the quasi-experimental design is adequate to measure impact. Large data sets are increasingly available in education, especially with the development of district and local data systems that expand the information gathered about students, teachers and schools. These data sets provide opportunities for research and evaluation studies that can operate with population level data rather than samples of data drawn randomly from the population. The determination of the quality of quasi-experimental evaluation designs that result from this study provides information that can be used by policy makers and researchers to study questions about educational treatments, programs and policies that have been intractable to study. Removing the potential barrier of random assignment, while still maintaining the quality of an RCT, expands the methodological toolkit of evaluators.

教育中的实验设计，包括随机分配到不同干预条件的治疗组和对照组，被认为是能够评估因果关系的最严格的选择。然而，随机对照试验 (RCT) 通常无法为许多 STEM 实践、计划和政策的影响提供证据。西北大学的研究人员研究了包括治疗组和对照组以及结果测量但不包括随机分配治疗和控制条件的准实验设计如何产生与实验设计相同质量的结果。研究人员还在研究是否可以用代理预测试来代替研究结果的预处理措施，即与原始预测试在同一领域但以不同形式进行的测量，以复制使用原始预测试的实验研究的结果。许多 STEM 研究经常在准实验研究中获取来自档案数据集中这些代理预测试的数据，例如州纵向数据系统。该项目的研究人员已经确定了许多随机对照试验，他们用这些随机对照试验来检验将对照组的性质从原始随机分配组更改为从较大人群进行统计调整的组的性质是否会显示与原始随机对照试验相似的因果估计。通过研究内比较，研究人员确定了随机对照试验和准实验结果的差异。他们正在研究代理预处理测试的使用，例如来自州纵向数据系统的数学成绩数据，以检查这些措施在多大程度上复制了原始随机对照试验中的结果。他们还研究了使用许多干预前协变量进行建模对因果估计的影响。如果随机对照试验和新的准实验研究中的因果估计之间的差异很小，那么准实验设计足以衡量影响。 教育领域越来越多地使用大数据集，特别是随着地区和地方数据系统的发展，扩大了收集的有关学生、教师和学校的信息。这些数据集为研究和评估研究提供了机会，这些研究和评估研究可以使用人口水平数据而不是从人口中随机抽取的数据样本进行操作。本研究确定的准实验评估设计的质量提供了政策制定者和研究人员可以用来研究难以研究的教育治疗、计划和政策问题的信息。消除随机分配的潜在障碍，同时仍然保持随机对照试验的质量，扩展了评估者的方法工具包。