Creating a Theoretical and Empirical Foundation for Better Non-Equivalent Control Group Designs in STEM Research

为 STEM 研究中更好的非等效对照组设计奠定理论和经验基础

• 批准号: 1760458
• 负责人: Thomas Cook
• 金额: $ 117.66万
• 依托单位: George Washington University
• 依托单位国家: 美国
• 项目类别: Standard Grant
• 财政年份: 2018
• 资助国家: 美国
• 起止时间: 2018-08-15 至 2022-07-31
• 项目状态: 已结题
• 来源: https://www.nsf.gov/awardsearch/showAward?AWD_ID=1760458&HistoricalAwards=false
• 关键词: Creating; Theoretical; Empirical; Foundation; Better

This proposal was submitted in response to EHR Core Research (ECR) program announcement NSF 15-509. The ECR program of fundamental research in STEM education provides funding in critical research areas that are essential, broad and enduring. EHR seeks proposals that will help synthesize, build and/or expand research foundations in the following focal areas: STEM learning, STEM learning environments, STEM workforce development, and broadening participation in STEM. The ECR program is distinguished by its emphasis on the accumulation of robust evidence to inform efforts to (a) understand, (b) build theory to explain, and (c) suggest interventions (and innovations) to address persistent challenges in STEM interest, education, learning, and participation. Random assignment of a STEM innovation to participants (students, teachers, schools) affords the researcher the opportunity to estimate an unbiased estimate of the effect of an intervention. However, random assignment may not be possible for ethical and practical reasons. Consequently, it is often difficult to build statistical knowledge of the impact of a STEM innovation without introducing bias into the process. The research to be conducted here will build and test a framework that will allow researchers to minimize statistical bias and better understand the impacts of STEM innovations (e.g., a new curriculum, a new way of teaching, or a change in policy) in settings where random assignment is not possible. This proposal uses advances in statistical theory and past empirical findings to craft testable hypotheses that, if validated, will improve causal hypothesis testing in STEM education research. The main hypotheses speak to reducing, and perhaps eliminating, the selection bias in quasi-experiments. The proposal focuses on the bias-reducing role of three design elements: a comparison group that is local to the treatment group, a pretest measure of the study outcome, and a rich set of pre-intervention covariates that are multi-dimensional (cover more than one substantive domain), multi-temporal (cover more than one time point) and multi-level (available at both the student and school levels). Each of these elements often reduces some bias and sometimes eliminates it entirely. How do they combine to reduce bias? The proposal will test (1) which combinations of the elements reduces most bias; (2) how often the bias reduces to close to zero; (3) do they do so robustly across STEM datasets with diverse interventions and students; and (4) is bias inadvertently introduced, rather than reduced, when the 3 design elements are combined.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.

该提案是为了响应 EHR 核心研究 (ECR) 计划公告 NSF 15-509 而提交的。 ECR STEM 教育基础研究计划为重要、广泛和持久的关键研究领域提供资金。 EHR 寻求有助于综合、建立和/或扩大以下重点领域研究基础的提案：STEM 学习、STEM 学习环境、STEM 劳动力发展以及扩大 STEM 的参与。 ECR 计划的特点是强调积累强有力的证据，为以下方面的努力提供信息：(a) 理解，(b) 建立理论来解释，以及 (c) 建议干预措施（和创新），以解决 STEM 兴趣、教育、学习和参与方面持续存在的挑战。将 STEM 创新随机分配给参与者（学生、教师、学校）使研究人员有机会对干预效果进行公正的估计。然而，出于道德和实际原因，随机分配可能是不可能的。因此，在不引入偏见的情况下，通常很难建立有关 STEM 创新影响的统计知识。这里要进行的研究将建立并测试一个框架，使研究人员能够最大限度地减少统计偏差，并更好地了解 STEM 创新（例如新课程、新教学方式或政策变化）在无法进行随机分配的环境中的影响。该提案利用统计理论的进步和过去的实证研究结果来制定可检验的假设，如果这些假设得到验证，将改善 STEM 教育研究中的因果假设检验。主要假设涉及减少甚至消除准实验中的选择偏差。该提案重点关注三个设计元素的减少偏差作用：治疗组本地的比较组、研究结果的预测试测量以及一组丰富的干预前协变量，这些协变量是多维度（涵盖多个实质性领域）、多时间（涵盖多个时间点）和多层次（在学生和学校层面均可用）。这些元素中的每一个通常都会减少一些偏见，有时甚至完全消除它。它们如何结合起来减少偏见？该提案将测试（1）哪些元素组合可以最大程度地减少偏差； (2) 偏差减少到接近于零的频率是多少； (3) 他们是否在具有不同干预措施和学生的 STEM 数据集上稳健地做到了这一点？ (4) 当这 3 个设计元素结合在一起时，无意中引入了偏见，而不是减少了偏见。该奖项反映了 NSF 的法定使命，并通过使用基金会的智力优点和更广泛的影响审查标准进行评估，被认为值得支持。