The Internal Validity of External Validity: Using Experiments to Validate Three Approaches to Extrapolating Causal Inferences Beyond the Cutoff in Regression Discontinuity

外部有效性的内部有效性：使用实验验证三种方法来推断超出回归不连续性截止值的因果推论

• 批准号: 1544301
• 负责人: Thomas Cook
• 金额: $ 79.46万
• 依托单位: MATHEMATICA POLICY RESEARCH INC
• 依托单位国家: 美国
• 项目类别: Standard Grant
• 财政年份: 2015
• 资助国家: 美国
• 起止时间: 2015-10-01 至 2018-12-31
• 项目状态: 已结题
• 来源: https://www.nsf.gov/awardsearch/showAward?AWD_ID=1544301&HistoricalAwards=false
• 关键词: Internal; Validity; External; Using; Experiments

The Promoting Research and Innovation in Methodologies for Evaluation (PRIME) program seeks to support research on evaluation with special emphasis on: (1) exploring innovative approaches for determining the impacts and usefulness of STEM education projects and programs; (2) building on and expanding the theoretical foundations for evaluating STEM education and workforce development initiatives, including translating and adapting approaches from other fields; and (3) growing the capacity and infrastructure of the evaluation field. Three types of proposals will be supported by the program: Exploratory Projects that include proof-of-concept and feasibility studies; more extensive Full-Scale Projects; and workshops and conferences. The proposed research attends carefully to item 1 above. In addition, the dissemination plan also attends to item 3. Randomized control trials in STEM education are considered the gold standard for causal inference. Often these models are overly restrictive in practice (for ethical and other reasons). Moreover, many such studies collapse as the research progresses. For example, senior school leadership changes and a school is withdrawn from a study. When this occurs the randomizing process is compromised and so are the resulting inferences. The proposed research will assess the validity and robustness of new approaches for using the regression discontinuity design (RDD) to evaluate educational interventions affecting STEM outcomes. This study will systematically compare and contrast the new approaches for generalizing RDD, assessing how well they produce unbiased causal estimates away from the cutoff score. The research will be conducted using 12 heterogeneous datasets from past educational interventions that (a) were originally evaluated through an RE and (b) included math achievement as an outcome measure. Using the within-study comparison method (Cook et al. 2008), this study will assess the extent to which the RE and a given RDD approach produce similar causal estimates away from the cutoff for the same population. Analyzing multiple datasets with STEM outcomes will provide information about the robustness of the new RDD methods across different types of interventions and student populations.

促进评估方法的研究和创新（PRIME）计划旨在支持评估研究，特别强调：（1）探索确定STEM教育项目和计划的影响和有用性的创新方法;（2）建立和扩大评估STEM教育和劳动力发展计划的理论基础，包括翻译和改编其他领域的方法;以及（3）增强评价领域的能力和基础设施。该计划将支持三种类型的提案：探索性项目，包括概念验证和可行性研究;更广泛的全面项目;以及研讨会和会议。拟议的研究仔细注意到上述第1项。此外，传播计划还涉及项目3。STEM教育中的随机对照试验被认为是因果推理的黄金标准。 这些模式在实践中往往过于严格（出于道德和其他原因）。此外，许多这样的研究随着研究的进展而崩溃。例如，高中领导层的变化和学校是从研究撤回。 当这种情况发生时，随机化过程会受到损害，由此产生的推论也会受到损害。拟议的研究将评估使用回归不连续设计（RDD）评估影响STEM结果的教育干预措施的新方法的有效性和稳健性。本研究将系统地比较和对比推广RDD的新方法，评估它们在多大程度上产生了偏离截止分数的无偏因果估计。这项研究将使用来自过去教育干预的12个异构数据集进行，这些数据集（a）最初是通过RE进行评估的，（B）包括数学成绩作为结果指标。使用研究内比较方法（Cook et al. 2008），本研究将评估RE和给定RDD方法在相同人群中产生相似因果估计值的程度。分析具有STEM结果的多个数据集将提供有关新RDD方法在不同类型的干预措施和学生群体中的稳健性的信息。