Creating a Theoretical and Empirical Foundation for Better Non-Equivalent Control Group Designs in STEM Research

该提案是根据EHR核心研究（ECR）项目公告NSF 15-509提交的。STEM教育基础研究ECR项目为重要、广泛和持久的关键研究领域提供资金。EHR寻求有助于在以下重点领域综合、建立和/或扩大研究基础的提案：STEM学习、STEM学习环境、STEM劳动力发展和扩大STEM参与。ECR项目的特点是强调积累有力的证据，为以下方面的努力提供信息：(a)理解，(b)建立理论来解释，(c)提出干预（和创新）措施，以解决STEM兴趣、教育、学习和参与方面的持续挑战。将STEM创新随机分配给参与者（学生、教师、学校），使研究人员有机会对干预措施的效果进行无偏估计。然而，出于道德和实际原因，随机分配可能是不可能的。因此，通常很难在不引入偏见的情况下建立STEM创新影响的统计知识。在这里进行的研究将建立和测试一个框架，该框架将使研究人员能够最大限度地减少统计偏差，并更好地理解STEM创新（例如，新课程，新教学方式或政策变化）在不可能随机分配的环境中的影响。该提案利用统计理论的进步和过去的实证研究结果来制定可测试的假设，如果得到验证，将改善STEM教育研究中的因果假设检验。主要的假设是减少，也许是消除准实验中的选择偏差。该建议侧重于三个设计元素的减少偏倚作用：治疗组本地的比较组，研究结果的预试测量，以及一组丰富的干预前协变量，这些协变量是多维的（覆盖多个实质性领域），多时间的（覆盖多个时间点）和多层次的（可在学生和学校层面使用）。这些元素中的每一个通常都会减少一些偏见，有时甚至完全消除偏见。它们是如何结合起来减少偏见的？该提案将测试(1)哪种元素组合能最大程度地减少偏见；(2)偏差减少到接近零的频率；(3)在不同干预措施和学生的STEM数据集中，他们是否做到了这一点？(4)当3个设计元素结合在一起时，是否无意中引入了偏差，而不是减少了偏差。该奖项反映了美国国家科学基金会的法定使命，并通过使用基金会的知识价值和更广泛的影响审查标准进行评估，被认为值得支持。

This proposal was submitted in response to EHR Core Research (ECR) program announcement NSF 15-509. The ECR program of fundamental research in STEM education provides funding in critical research areas that are essential, broad and enduring. EHR seeks proposals that will help synthesize, build and/or expand research foundations in the following focal areas: STEM learning, STEM learning environments, STEM workforce development, and broadening participation in STEM. The ECR program is distinguished by its emphasis on the accumulation of robust evidence to inform efforts to (a) understand, (b) build theory to explain, and (c) suggest interventions (and innovations) to address persistent challenges in STEM interest, education, learning, and participation. Random assignment of a STEM innovation to participants (students, teachers, schools) affords the researcher the opportunity to estimate an unbiased estimate of the effect of an intervention. However, random assignment may not be possible for ethical and practical reasons. Consequently, it is often difficult to build statistical knowledge of the impact of a STEM innovation without introducing bias into the process. The research to be conducted here will build and test a framework that will allow researchers to minimize statistical bias and better understand the impacts of STEM innovations (e.g., a new curriculum, a new way of teaching, or a change in policy) in settings where random assignment is not possible. This proposal uses advances in statistical theory and past empirical findings to craft testable hypotheses that, if validated, will improve causal hypothesis testing in STEM education research. The main hypotheses speak to reducing, and perhaps eliminating, the selection bias in quasi-experiments. The proposal focuses on the bias-reducing role of three design elements: a comparison group that is local to the treatment group, a pretest measure of the study outcome, and a rich set of pre-intervention covariates that are multi-dimensional (cover more than one substantive domain), multi-temporal (cover more than one time point) and multi-level (available at both the student and school levels). Each of these elements often reduces some bias and sometimes eliminates it entirely. How do they combine to reduce bias? The proposal will test (1) which combinations of the elements reduces most bias; (2) how often the bias reduces to close to zero; (3) do they do so robustly across STEM datasets with diverse interventions and students; and (4) is bias inadvertently introduced, rather than reduced, when the 3 design elements are combined.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.