Creating a Theoretical and Empirical Foundation for Better Non-Equivalent Control Group Designs in STEM Research

该提案是为了响应EHR核心研究（ECR）计划公告NSF 15-509而提交的。在STEM教育的基础研究ECR计划提供资金在关键的研究领域是必不可少的，广泛的和持久的。EHR寻求有助于综合，建立和/或扩大以下重点领域研究基础的建议：STEM学习，STEM学习环境，STEM劳动力发展和扩大STEM参与。ECR计划的特点是强调积累强有力的证据，以（a）理解，（B）建立理论来解释，（c）提出干预措施（和创新），以解决STEM兴趣，教育，学习和参与方面的持续挑战。将STEM创新随机分配给参与者（学生，教师，学校），研究人员有机会对干预措施的效果进行无偏估计。然而，出于道德和实际原因，随机分配可能是不可能的。因此，在不引入偏见的情况下，通常很难建立STEM创新影响的统计知识。在这里进行的研究将建立和测试一个框架，使研究人员能够最大限度地减少统计偏差，更好地了解STEM创新的影响（例如，新的课程、新的教学方式或政策的改变）。该提案利用统计理论的进步和过去的实证研究结果来制定可检验的假设，如果得到验证，将改善STEM教育研究中的因果假设检验。主要的假设是减少，甚至消除准实验中的选择偏差。该提案侧重于三个设计要素的偏倚减少作用：治疗组本地的比较组，研究结果的预测试措施，以及一组丰富的干预前协变量，这些协变量是多维的（涵盖一个以上的实质性领域），多时间（涵盖一个以上的时间点）和多水平（可在学生和学校两级）。这些因素中的每一个往往会减少一些偏见，有时甚至完全消除偏见。它们如何联合收割机来减少偏差？该提案将测试（1）哪些元素的组合减少了最大的偏差;（2）偏差减少到接近零的频率;（3）在具有不同干预措施和学生的STEM数据集上，它们是否能够稳健地做到这一点;（4）是无意中引入的偏差，而不是减少的，该奖项反映了NSF的法定使命，并通过使用基金会的智力价值进行评估，更广泛的影响审查标准。

This proposal was submitted in response to EHR Core Research (ECR) program announcement NSF 15-509. The ECR program of fundamental research in STEM education provides funding in critical research areas that are essential, broad and enduring. EHR seeks proposals that will help synthesize, build and/or expand research foundations in the following focal areas: STEM learning, STEM learning environments, STEM workforce development, and broadening participation in STEM. The ECR program is distinguished by its emphasis on the accumulation of robust evidence to inform efforts to (a) understand, (b) build theory to explain, and (c) suggest interventions (and innovations) to address persistent challenges in STEM interest, education, learning, and participation. Random assignment of a STEM innovation to participants (students, teachers, schools) affords the researcher the opportunity to estimate an unbiased estimate of the effect of an intervention. However, random assignment may not be possible for ethical and practical reasons. Consequently, it is often difficult to build statistical knowledge of the impact of a STEM innovation without introducing bias into the process. The research to be conducted here will build and test a framework that will allow researchers to minimize statistical bias and better understand the impacts of STEM innovations (e.g., a new curriculum, a new way of teaching, or a change in policy) in settings where random assignment is not possible. This proposal uses advances in statistical theory and past empirical findings to craft testable hypotheses that, if validated, will improve causal hypothesis testing in STEM education research. The main hypotheses speak to reducing, and perhaps eliminating, the selection bias in quasi-experiments. The proposal focuses on the bias-reducing role of three design elements: a comparison group that is local to the treatment group, a pretest measure of the study outcome, and a rich set of pre-intervention covariates that are multi-dimensional (cover more than one substantive domain), multi-temporal (cover more than one time point) and multi-level (available at both the student and school levels). Each of these elements often reduces some bias and sometimes eliminates it entirely. How do they combine to reduce bias? The proposal will test (1) which combinations of the elements reduces most bias; (2) how often the bias reduces to close to zero; (3) do they do so robustly across STEM datasets with diverse interventions and students; and (4) is bias inadvertently introduced, rather than reduced, when the 3 design elements are combined.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.